X-Git-Url: https://git.rapsys.eu/youtubedl/blobdiff_plain/b238854ce845f3796daac74edab2e8a373e8ba1a..6245a4128588146e39f66b371624ee39bdbb7720:/README.txt diff --git a/README.txt b/README.txt index fc9f17e..cc86a1b 100644 --- a/README.txt +++ b/README.txt @@ -1,248 +1,708 @@ -NAME -==== +[Build Status] youtube-dl - download videos from youtube.com or other video platforms -SYNOPSIS -======== +- INSTALLATION +- DESCRIPTION +- OPTIONS +- CONFIGURATION +- OUTPUT TEMPLATE +- FORMAT SELECTION +- VIDEO SELECTION +- FAQ +- DEVELOPER INSTRUCTIONS +- EMBEDDING YOUTUBE-DL +- BUGS +- COPYRIGHT + + + +INSTALLATION + + +To install it right away for all UNIX users (Linux, macOS, etc.), type: + + sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl + sudo chmod a+rx /usr/local/bin/youtube-dl + +If you do not have curl, you can alternatively use a recent wget: + + sudo wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl + sudo chmod a+rx /usr/local/bin/youtube-dl + +Windows users can download an .exe file and place it in any location on +their PATH except for %SYSTEMROOT%\System32 (e.g. DO NOT put in +C:\Windows\System32). + +You can also use pip: + + sudo -H pip install --upgrade youtube-dl + +This command will update youtube-dl if you have already installed it. +See the pypi page for more information. + +macOS users can install youtube-dl with Homebrew: + + brew install youtube-dl + +Or with MacPorts: + + sudo port install youtube-dl + +Alternatively, refer to the developer instructions for how to check out +and work with the git repository. For further options, including PGP +signatures, see the youtube-dl Download Page. + -youtube-dl OPTIONS URL [URL...] DESCRIPTION -=========== -youtube-dl is a small command-line program to download videos from -YouTube.com and a few more sites. It requires the Python interpreter, -version 2.6, 2.7, or 3.3+, and it is not platform specific. It should -work on your Unix box, on Windows or on Mac OS X. It is released to the -public domain, which means you can modify it, redistribute it or use it -however you like. + +YOUTUBE-DL is a command-line program to download videos from YouTube.com +and a few more sites. It requires the Python interpreter, version 2.6, +2.7, or 3.2+, and it is not platform specific. It should work on your +Unix box, on Windows or on macOS. It is released to the public domain, +which means you can modify it, redistribute it or use it however you +like. + + youtube-dl [OPTIONS] URL [URL...] + + OPTIONS -======= - - -h, --help print this help text and exit - --version print program version and exit - -U, --update update this program to latest version. Make sure - that you have sufficient permissions (run with - sudo if needed) - -i, --ignore-errors continue on download errors, for example to to - skip unavailable videos in a playlist - --abort-on-error Abort downloading of further videos (in the - playlist or the command line) if an error occurs - --dump-user-agent display the current browser identification - --user-agent UA specify a custom user agent - --referer REF specify a custom referer, use if the video access - is restricted to one domain - --list-extractors List all supported extractors and the URLs they - would handle - --extractor-descriptions Output descriptions of all supported extractors - --proxy URL Use the specified HTTP/HTTPS proxy. Pass in an - empty string (--proxy "") for direct connection - --no-check-certificate Suppress HTTPS certificate validation. - --cache-dir DIR Location in the filesystem where youtube-dl can - store downloaded information permanently. By - default $XDG_CACHE_HOME/youtube-dl or ~/.cache - /youtube-dl . - --no-cache-dir Disable filesystem caching + + + -h, --help Print this help text and exit + --version Print program version and exit + -U, --update Update this program to latest version. Make + sure that you have sufficient permissions + (run with sudo if needed) + -i, --ignore-errors Continue on download errors, for example to + skip unavailable videos in a playlist + --abort-on-error Abort downloading of further videos (in the + playlist or the command line) if an error + occurs + --dump-user-agent Display the current browser identification + --list-extractors List all supported extractors + --extractor-descriptions Output descriptions of all supported + extractors + --force-generic-extractor Force extraction to use the generic + extractor + --default-search PREFIX Use this prefix for unqualified URLs. For + example "gvsearch2:" downloads two videos + from google videos for youtube-dl "large + apple". Use the value "auto" to let + youtube-dl guess ("auto_warning" to emit a + warning when guessing). "error" just throws + an error. The default value "fixup_error" + repairs broken URLs, but emits an error if + this is not possible instead of searching. + --ignore-config Do not read configuration files. When given + in the global configuration file + /etc/youtube-dl.conf: Do not read the user + configuration in ~/.config/youtube- + dl/config (%APPDATA%/youtube-dl/config.txt + on Windows) + --config-location PATH Location of the configuration file; either + the path to the config or its containing + directory. + --flat-playlist Do not extract the videos of a playlist, + only list them. + --mark-watched Mark videos watched (YouTube only) + --no-mark-watched Do not mark videos watched (YouTube only) + --no-color Do not emit color codes in output + + +Network Options: + + --proxy URL Use the specified HTTP/HTTPS/SOCKS proxy. + To enable SOCKS proxy, specify a proper + scheme. For example + socks5://127.0.0.1:1080/. Pass in an empty + string (--proxy "") for direct connection + --socket-timeout SECONDS Time to wait before giving up, in seconds + --source-address IP Client-side IP address to bind to + -4, --force-ipv4 Make all connections via IPv4 + -6, --force-ipv6 Make all connections via IPv6 + + +Geo Restriction: + + --geo-verification-proxy URL Use this proxy to verify the IP address for + some geo-restricted sites. The default + proxy specified by --proxy (or none, if the + option is not present) is used for the + actual downloading. + --geo-bypass Bypass geographic restriction via faking + X-Forwarded-For HTTP header + --no-geo-bypass Do not bypass geographic restriction via + faking X-Forwarded-For HTTP header + --geo-bypass-country CODE Force bypass geographic restriction with + explicitly provided two-letter ISO 3166-2 + country code + --geo-bypass-ip-block IP_BLOCK Force bypass geographic restriction with + explicitly provided IP block in CIDR + notation + Video Selection: ----------------- - - --playlist-start NUMBER playlist video to start at (default is 1) - --playlist-end NUMBER playlist video to end at (default is last) - --match-title REGEX download only matching titles (regex or caseless - sub-string) - --reject-title REGEX skip download for matching titles (regex or - caseless sub-string) - --max-downloads NUMBER Abort after downloading NUMBER files - --min-filesize SIZE Do not download any videos smaller than SIZE - (e.g. 50k or 44.6m) - --max-filesize SIZE Do not download any videos larger than SIZE (e.g. - 50k or 44.6m) - --date DATE download only videos uploaded in this date - --datebefore DATE download only videos uploaded before this date - --dateafter DATE download only videos uploaded after this date - --no-playlist download only the currently playing video - --age-limit YEARS download only videos suitable for the given age - --download-archive FILE Download only videos not listed in the archive - file. Record the IDs of all downloaded videos in - it. + + --playlist-start NUMBER Playlist video to start at (default is 1) + --playlist-end NUMBER Playlist video to end at (default is last) + --playlist-items ITEM_SPEC Playlist video items to download. Specify + indices of the videos in the playlist + separated by commas like: "--playlist-items + 1,2,5,8" if you want to download videos + indexed 1, 2, 5, 8 in the playlist. You can + specify range: "--playlist-items + 1-3,7,10-13", it will download the videos + at index 1, 2, 3, 7, 10, 11, 12 and 13. + --match-title REGEX Download only matching titles (regex or + caseless sub-string) + --reject-title REGEX Skip download for matching titles (regex or + caseless sub-string) + --max-downloads NUMBER Abort after downloading NUMBER files + --min-filesize SIZE Do not download any videos smaller than + SIZE (e.g. 50k or 44.6m) + --max-filesize SIZE Do not download any videos larger than SIZE + (e.g. 50k or 44.6m) + --date DATE Download only videos uploaded in this date + --datebefore DATE Download only videos uploaded on or before + this date (i.e. inclusive) + --dateafter DATE Download only videos uploaded on or after + this date (i.e. inclusive) + --min-views COUNT Do not download any videos with less than + COUNT views + --max-views COUNT Do not download any videos with more than + COUNT views + --match-filter FILTER Generic video filter. Specify any key (see + the "OUTPUT TEMPLATE" for a list of + available keys) to match if the key is + present, !key to check if the key is not + present, key > NUMBER (like "comment_count + > 12", also works with >=, <, <=, !=, =) to + compare against a number, key = 'LITERAL' + (like "uploader = 'Mike Smith'", also works + with !=) to match against a string literal + and & to require multiple matches. Values + which are not known are excluded unless you + put a question mark (?) after the operator. + For example, to only match videos that have + been liked more than 100 times and disliked + less than 50 times (or the dislike + functionality is not available at the given + service), but who also have a description, + use --match-filter "like_count > 100 & + dislike_count .+?) - (?P.+)" + --xattrs Write metadata to the video file's xattrs + (using dublin core and xdg standards) + --fixup POLICY Automatically correct known faults of the + file. One of never (do nothing), warn (only + emit a warning), detect_or_warn (the + default; fix file if we can, warn + otherwise) + --prefer-avconv Prefer avconv over ffmpeg for running the + postprocessors + --prefer-ffmpeg Prefer ffmpeg over avconv for running the + postprocessors (default) + --ffmpeg-location PATH Location of the ffmpeg/avconv binary; + either the path to the binary or its + containing directory. + --exec CMD Execute a command on the file after + downloading, similar to find's -exec + syntax. Example: --exec 'adb push {} + /sdcard/Music/ && rm {}' + --convert-subs FORMAT Convert the subtitles to other format + (currently supported: srt|ass|vtt|lrc) + + CONFIGURATION -============= -You can configure youtube-dl by placing default arguments (such as ---extract-audio --no-mtime to always extract the audio and not copy the -mtime) into /etc/youtube-dl.conf and/or ~/.config/youtube-dl.conf. On -Windows, the configuration file locations are -%APPDATA%\youtube-dl\config.txt and C:\Users\<Yourname>\youtube-dl.conf. + +You can configure youtube-dl by placing any supported command line +option to a configuration file. On Linux and macOS, the system wide +configuration file is located at /etc/youtube-dl.conf and the user wide +configuration file at ~/.config/youtube-dl/config. On Windows, the user +wide configuration file locations are %APPDATA%\youtube-dl\config.txt or +C:\Users\<user name>\youtube-dl.conf. Note that by default configuration +file may not exist so you may need to create it yourself. + +For example, with the following configuration file youtube-dl will +always extract the audio, not copy the mtime, use a proxy and save all +videos under Movies directory in your home directory: + + # Lines starting with # are comments + + # Always extract audio + -x + + # Do not copy the mtime + --no-mtime + + # Use this proxy + --proxy 127.0.0.1:3128 + + # Save all videos under Movies directory in your home directory + -o ~/Movies/%(title)s.%(ext)s + +Note that options in configuration file are just the same options aka +switches used in regular command line calls thus there MUST BE NO +WHITESPACE after - or --, e.g. -o or --proxy but not - o or -- proxy. + +You can use --ignore-config if you want to disable the configuration +file for a particular youtube-dl run. + +You can also use --config-location if you want to use custom +configuration file for a particular youtube-dl run. + +Authentication with .netrc file + +You may also want to configure automatic credentials storage for +extractors that support authentication (by providing login and password +with --username and --password) in order not to pass credentials as +command line arguments on every youtube-dl execution and prevent +tracking plain text passwords in the shell command history. You can +achieve this using a .netrc file on a per extractor basis. For that you +will need to create a .netrc file in your $HOME and restrict permissions +to read/write by only you: + + touch $HOME/.netrc + chmod a-rwx,u+rw $HOME/.netrc + +After that you can add credentials for an extractor in the following +format, where _extractor_ is the name of the extractor in lowercase: + + machine <extractor> login <login> password <password> + +For example: + + machine youtube login myaccount@gmail.com password my_youtube_password + machine twitch login my_twitch_account_name password my_twitch_password + +To activate authentication with the .netrc file you should pass --netrc +to youtube-dl or place it in the configuration file. + +On Windows you may also need to setup the %HOME% environment variable +manually. For example: + + set HOME=%USERPROFILE% + + OUTPUT TEMPLATE -=============== + The -o option allows users to indicate a template for the output file -names. The basic usage is not to set any template arguments when -downloading a single file, like in -youtube-dl -o funny_video.flv "http://some/video". However, it may -contain special sequences that will be replaced when downloading each -video. The special sequences have the format %(NAME)s. To clarify, that -is a percent symbol followed by a name in parenthesis, followed by a -lowercase S. Allowed names are: - -- id: The sequence will be replaced by the video identifier. -- url: The sequence will be replaced by the video URL. -- uploader: The sequence will be replaced by the nickname of the - person who uploaded the video. -- upload_date: The sequence will be replaced by the upload date in - YYYYMMDD format. -- title: The sequence will be replaced by the video title. -- ext: The sequence will be replaced by the appropriate extension - (like flv or mp4). -- epoch: The sequence will be replaced by the Unix epoch when creating - the file. -- autonumber: The sequence will be replaced by a five-digit number - that will be increased with each download, starting at zero. -- playlist: The name or the id of the playlist that contains the - video. -- playlist_index: The index of the video in the playlist, a five-digit - number. +names. + +TL;DR: navigate me to examples. + +The basic usage is not to set any template arguments when downloading a +single file, like in youtube-dl -o funny_video.flv "https://some/video". +However, it may contain special sequences that will be replaced when +downloading each video. The special sequences may be formatted according +to python string formatting operations. For example, %(NAME)s or +%(NAME)05d. To clarify, that is a percent symbol followed by a name in +parentheses, followed by formatting operations. Allowed names along with +sequence type are: + +- id (string): Video identifier +- title (string): Video title +- url (string): Video URL +- ext (string): Video filename extension +- alt_title (string): A secondary title of the video +- display_id (string): An alternative identifier for the video +- uploader (string): Full name of the video uploader +- license (string): License name the video is licensed under +- creator (string): The creator of the video +- release_date (string): The date (YYYYMMDD) when the video was + released +- timestamp (numeric): UNIX timestamp of the moment the video became + available +- upload_date (string): Video upload date (YYYYMMDD) +- uploader_id (string): Nickname or id of the video uploader +- channel (string): Full name of the channel the video is uploaded on +- channel_id (string): Id of the channel +- location (string): Physical location where the video was filmed +- duration (numeric): Length of the video in seconds +- view_count (numeric): How many users have watched the video on the + platform +- like_count (numeric): Number of positive ratings of the video +- dislike_count (numeric): Number of negative ratings of the video +- repost_count (numeric): Number of reposts of the video +- average_rating (numeric): Average rating give by users, the scale + used depends on the webpage +- comment_count (numeric): Number of comments on the video +- age_limit (numeric): Age restriction for the video (years) +- is_live (boolean): Whether this video is a live stream or a + fixed-length video +- start_time (numeric): Time in seconds where the reproduction should + start, as specified in the URL +- end_time (numeric): Time in seconds where the reproduction should + end, as specified in the URL +- format (string): A human-readable description of the format +- format_id (string): Format code specified by --format +- format_note (string): Additional info about the format +- width (numeric): Width of the video +- height (numeric): Height of the video +- resolution (string): Textual description of width and height +- tbr (numeric): Average bitrate of audio and video in KBit/s +- abr (numeric): Average audio bitrate in KBit/s +- acodec (string): Name of the audio codec in use +- asr (numeric): Audio sampling rate in Hertz +- vbr (numeric): Average video bitrate in KBit/s +- fps (numeric): Frame rate +- vcodec (string): Name of the video codec in use +- container (string): Name of the container format +- filesize (numeric): The number of bytes, if known in advance +- filesize_approx (numeric): An estimate for the number of bytes +- protocol (string): The protocol that will be used for the actual + download +- extractor (string): Name of the extractor +- extractor_key (string): Key name of the extractor +- epoch (numeric): Unix epoch when creating the file +- autonumber (numeric): Five-digit number that will be increased with + each download, starting at zero +- playlist (string): Name or id of the playlist that contains the + video +- playlist_index (numeric): Index of the video in the playlist padded + with leading zeros according to the total length of the playlist +- playlist_id (string): Playlist identifier +- playlist_title (string): Playlist title +- playlist_uploader (string): Full name of the playlist uploader +- playlist_uploader_id (string): Nickname or id of the playlist + uploader + +Available for the video that belongs to some logical chapter or section: + +- chapter (string): Name or title of the chapter the video belongs to +- chapter_number (numeric): Number of the chapter the video belongs to +- chapter_id (string): Id of the chapter the video belongs to + +Available for the video that is an episode of some series or programme: + +- series (string): Title of the series or programme the video episode + belongs to +- season (string): Title of the season the video episode belongs to +- season_number (numeric): Number of the season the video episode + belongs to +- season_id (string): Id of the season the video episode belongs to +- episode (string): Title of the video episode +- episode_number (numeric): Number of the video episode within a + season +- episode_id (string): Id of the video episode + +Available for the media that is a track or a part of a music album: + +- track (string): Title of the track +- track_number (numeric): Number of the track within an album or a + disc +- track_id (string): Id of the track +- artist (string): Artist(s) of the track +- genre (string): Genre(s) of the track +- album (string): Title of the album the track belongs to +- album_type (string): Type of the album +- album_artist (string): List of all artists appeared on the album +- disc_number (numeric): Number of the disc or other physical medium + the track belongs to +- release_year (numeric): Year (YYYY) when the album was released + +Each aforementioned sequence when referenced in an output template will +be replaced by the actual value corresponding to the sequence name. Note +that some of the sequences are not guaranteed to be present since they +depend on the metadata obtained by a particular extractor. Such +sequences will be replaced with NA. + +For example for -o %(title)s-%(id)s.%(ext)s and an mp4 video with title +youtube-dl test video and id BaW_jenozKcj, this will result in a +youtube-dl test video-BaW_jenozKcj.mp4 file created in the current +directory. + +For numeric sequences you can use numeric related formatting, for +example, %(view_count)05d will result in a string with view count padded +with zeros up to 5 characters, like in 00042. + +Output templates can also contain arbitrary hierarchical path, e.g. +-o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' which will +result in downloading each video in a directory corresponding to this +path template. Any missing directory will be automatically created for +you. + +To use percent literals in an output template use %%. To output to +stdout use -o -. The current default template is %(title)s-%(id)s.%(ext)s. @@ -251,16 +711,209 @@ In some cases, you don't want special characters such as 中, spaces, or or the filename through an 8bit-unsafe channel. In these cases, add the --restrict-filenames flag to get a shorter title: - $ youtube-dl --get-filename -o "%(title)s.%(ext)s" BaW_jenozKc +Output template and Windows batch files + +If you are using an output template inside a Windows batch file then you +must escape plain percent characters (%) by doubling, so that +-o "%(title)s-%(id)s.%(ext)s" should become +-o "%%(title)s-%%(id)s.%%(ext)s". However you should not touch %'s that +are not plain characters, e.g. environment variables for expansion +should stay intact: -o "C:\%HOMEPATH%\Desktop\%%(title)s.%%(ext)s". + +Output template examples + +Note that on Windows you may need to use double quotes instead of +single. + + $ youtube-dl --get-filename -o '%(title)s.%(ext)s' BaW_jenozKc youtube-dl test video ''_ä↭𝕐.mp4 # All kinds of weird characters - $ youtube-dl --get-filename -o "%(title)s.%(ext)s" BaW_jenozKc --restrict-filenames + + $ youtube-dl --get-filename -o '%(title)s.%(ext)s' BaW_jenozKc --restrict-filenames youtube-dl_test_video_.mp4 # A simple file name + # Download YouTube playlist videos in separate directory indexed by video order in a playlist + $ youtube-dl -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re + + # Download all playlists of YouTube channel/user keeping each playlist in separate directory: + $ youtube-dl -o '%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/user/TheLinuxFoundation/playlists + + # Download Udemy course keeping each chapter in separate directory under MyVideos directory in your home + $ youtube-dl -u user -p password -o '~/MyVideos/%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s' https://www.udemy.com/java-tutorial/ + + # Download entire series season keeping each series and each season in separate directory under C:/MyVideos + $ youtube-dl -o "C:/MyVideos/%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" https://videomore.ru/kino_v_detalayah/5_sezon/367617 + + # Stream the video being downloaded to stdout + $ youtube-dl -o - BaW_jenozKc + + + +FORMAT SELECTION + + +By default youtube-dl tries to download the best available quality, i.e. +if you want the best quality you DON'T NEED to pass any special options, +youtube-dl will guess it for you by DEFAULT. + +But sometimes you may want to download in a different format, for +example when you are on a slow or intermittent connection. The key +mechanism for achieving this is so-called _format selection_ based on +which you can explicitly specify desired format, select formats based on +some criterion or criteria, setup precedence and much more. + +The general syntax for format selection is --format FORMAT or shorter +-f FORMAT where FORMAT is a _selector expression_, i.e. an expression +that describes format or formats you would like to download. + +TL;DR: navigate me to examples. + +The simplest case is requesting a specific format, for example with +-f 22 you can download the format with format code equal to 22. You can +get the list of available format codes for particular video using +--list-formats or -F. Note that these format codes are extractor +specific. + +You can also use a file extension (currently 3gp, aac, flv, m4a, mp3, +mp4, ogg, wav, webm are supported) to download the best quality format +of a particular file extension served as a single file, e.g. -f webm +will download the best quality format with the webm extension served as +a single file. + +You can also use special names to select particular edge case formats: + +- best: Select the best quality format represented by a single file + with video and audio. +- worst: Select the worst quality format represented by a single file + with video and audio. +- bestvideo: Select the best quality video-only format (e.g. DASH + video). May not be available. +- worstvideo: Select the worst quality video-only format. May not be + available. +- bestaudio: Select the best quality audio only-format. May not be + available. +- worstaudio: Select the worst quality audio only-format. May not be + available. + +For example, to download the worst quality video-only format you can use +-f worstvideo. + +If you want to download multiple videos and they don't have the same +formats available, you can specify the order of preference using +slashes. Note that slash is left-associative, i.e. formats on the left +hand side are preferred, for example -f 22/17/18 will download format 22 +if it's available, otherwise it will download format 17 if it's +available, otherwise it will download format 18 if it's available, +otherwise it will complain that no suitable formats are available for +download. + +If you want to download several formats of the same video use a comma as +a separator, e.g. -f 22,17,18 will download all these three formats, of +course if they are available. Or a more sophisticated example combined +with the precedence feature: -f 136/137/mp4/bestvideo,140/m4a/bestaudio. + +You can also filter the video formats by putting a condition in +brackets, as in -f "best[height=720]" (or -f "[filesize>10M]"). + +The following numeric meta fields can be used with comparisons <, <=, >, +>=, = (equals), != (not equals): + +- filesize: The number of bytes, if known in advance +- width: Width of the video, if known +- height: Height of the video, if known +- tbr: Average bitrate of audio and video in KBit/s +- abr: Average audio bitrate in KBit/s +- vbr: Average video bitrate in KBit/s +- asr: Audio sampling rate in Hertz +- fps: Frame rate + +Also filtering work for comparisons = (equals), ^= (starts with), $= +(ends with), *= (contains) and following string meta fields: + +- ext: File extension +- acodec: Name of the audio codec in use +- vcodec: Name of the video codec in use +- container: Name of the container format +- protocol: The protocol that will be used for the actual download, + lower-case (http, https, rtsp, rtmp, rtmpe, mms, f4m, ism, + http_dash_segments, m3u8, or m3u8_native) +- format_id: A short description of the format + +Any string comparison may be prefixed with negation ! in order to +produce an opposite comparison, e.g. !*= (does not contain). + +Note that none of the aforementioned meta fields are guaranteed to be +present since this solely depends on the metadata obtained by particular +extractor, i.e. the metadata offered by the video hoster. + +Formats for which the value is not known are excluded unless you put a +question mark (?) after the operator. You can combine format filters, so +-f "[height <=? 720][tbr>500]" selects up to 720p videos (or videos +where the height is not known) with a bitrate of at least 500 KBit/s. + +You can merge the video and audio of two formats into a single file +using -f <video-format>+<audio-format> (requires ffmpeg or avconv +installed), for example -f bestvideo+bestaudio will download the best +video-only format, the best audio-only format and mux them together with +ffmpeg/avconv. + +Format selectors can also be grouped using parentheses, for example if +you want to download the best mp4 and webm formats with a height lower +than 480 you can use -f '(mp4,webm)[height<480]'. + +Since the end of April 2015 and version 2015.04.26, youtube-dl uses +-f bestvideo+bestaudio/best as the default format selection (see #5447, +#5456). If ffmpeg or avconv are installed this results in downloading +bestvideo and bestaudio separately and muxing them together into a +single file giving the best overall quality available. Otherwise it +falls back to best and results in downloading the best available quality +served as a single file. best is also needed for videos that don't come +from YouTube because they don't provide the audio and video in two +different files. If you want to only download some DASH formats (for +example if you are not interested in getting videos with a resolution +higher than 1080p), you can add +-f bestvideo[height<=?1080]+bestaudio/best to your configuration file. +Note that if you use youtube-dl to stream to stdout (and most likely to +pipe it to your media player then), i.e. you explicitly specify output +template as -o -, youtube-dl still uses -f best format selection in +order to start content delivery immediately to your player and not to +wait until bestvideo and bestaudio are downloaded and muxed. + +If you want to preserve the old format selection behavior (prior to +youtube-dl 2015.04.26), i.e. you want to download the best available +quality media served as a single file, you should explicitly specify +your choice with -f best. You may want to add it to the configuration +file in order not to type it every time you run youtube-dl. + +Format selection examples + +Note that on Windows you may need to use double quotes instead of +single. + + # Download best mp4 format available or any other best if no mp4 available + $ youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' + + # Download best format available but no better than 480p + $ youtube-dl -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' + + # Download best video only format but no bigger than 50 MB + $ youtube-dl -f 'best[filesize<50M]' + + # Download best format available via direct link over HTTP/HTTPS protocol + $ youtube-dl -f '(bestvideo+bestaudio/best)[protocol^=http]' + + # Download the best video format and the best audio format without merging them + $ youtube-dl -f 'bestvideo,bestaudio' -o '%(title)s.f%(format_id)s.%(ext)s' + +Note that in the last example, an output template is recommended as +bestvideo and bestaudio may have the same file name. + + + VIDEO SELECTION -=============== + Videos can be filtered by their upload date using the options --date, ---datebefore or --dateafter, they accept dates in two formats: +--datebefore or --dateafter. They accept dates in two formats: - Absolute dates: Dates in the format YYYYMMDD. - Relative dates: Dates in the format @@ -268,12 +921,81 @@ Videos can be filtered by their upload date using the options --date, Examples: - $ youtube-dl --dateafter now-6months #will only download the videos uploaded in the last 6 months - $ youtube-dl --date 19700101 #will only download the videos uploaded in January 1, 1970 - $ youtube-dl --dateafter 20000101 --datebefore 20100101 #will only download the videos uploaded between 2000 and 2010 + # Download only the videos uploaded in the last 6 months + $ youtube-dl --dateafter now-6months + + # Download only the videos uploaded on January 1, 1970 + $ youtube-dl --date 19700101 + + $ # Download only the videos uploaded in the 200x decade + $ youtube-dl --dateafter 20000101 --datebefore 20091231 + + FAQ -=== + + +How do I update youtube-dl? + +If you've followed our manual installation instructions, you can simply +run youtube-dl -U (or, on Linux, sudo youtube-dl -U). + +If you have used pip, a simple sudo pip install -U youtube-dl is +sufficient to update. + +If you have installed youtube-dl using a package manager like _apt-get_ +or _yum_, use the standard system update mechanism to update. Note that +distribution packages are often outdated. As a rule of thumb, youtube-dl +releases at least once a month, and often weekly or even daily. Simply +go to https://yt-dl.org to find out the current version. Unfortunately, +there is nothing we youtube-dl developers can do if your distribution +serves a really outdated version. You can (and should) complain to your +distribution in their bugtracker or support forum. + +As a last resort, you can also uninstall the version installed by your +package manager and follow our manual installation instructions. For +that, remove the distribution's package, with a line like + + sudo apt-get remove -y youtube-dl + +Afterwards, simply follow our manual installation instructions: + + sudo wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl + sudo chmod a+rx /usr/local/bin/youtube-dl + hash -r + +Again, from then on you'll be able to update with sudo youtube-dl -U. + +youtube-dl is extremely slow to start on Windows + +Add a file exclusion for youtube-dl.exe in Windows Defender settings. + +I'm getting an error Unable to extract OpenGraph title on YouTube playlists + +YouTube changed their playlist format in March 2014 and later on, so +you'll need at least youtube-dl 2014.07.25 to download all YouTube +videos. + +If you have installed youtube-dl with a package manager, pip, setup.py +or a tarball, please use that to update. Note that Ubuntu packages do +not seem to get updated anymore. Since we are not affiliated with +Ubuntu, there is little we can do. Feel free to report bugs to the +Ubuntu packaging people - all they have to do is update the package to a +somewhat recent version. See above for a way to update. + +I'm getting an error when trying to use output template: error: using output template conflicts with using title, video ID or auto number + +Make sure you are not using -o with any of these options -t, --title, +--id, -A or --auto-number set in command line or in a configuration +file. Remove the latter if any. + +Do I always have to pass -citw? + +By default, youtube-dl intends to have the best options (incidentally, +if you have a convincing case that these should be different, please +file an issue where you explain that). Therefore, it is unnecessary and +sometimes harmful to copy long option strings from webpages. In +particular, the only option out of -citw that is regularly useful is -i. Can you please put the -b option back? @@ -289,34 +1011,100 @@ I get HTTP error 402 when trying to download a video. What's this? Apparently YouTube requires you to pass a CAPTCHA test if you download too much. We're considering to provide a way to let you solve the -CAPTCHA, but at the moment, your best course of action is pointing a -webbrowser to the youtube URL, solving the CAPTCHA, and restart -youtube-dl. +CAPTCHA, but at the moment, your best course of action is pointing a web +browser to the youtube URL, solving the CAPTCHA, and restart youtube-dl. + +Do I need any other programs? + +youtube-dl works fine on its own on most sites. However, if you want to +convert video/audio, you'll need avconv or ffmpeg. On some sites - most +notably YouTube - videos can be retrieved in a higher quality format +without sound. youtube-dl will detect whether avconv/ffmpeg is present +and automatically pick the best option. + +Videos or video formats streamed via RTMP protocol can only be +downloaded when rtmpdump is installed. Downloading MMS and RTSP videos +requires either mplayer or mpv to be installed. I have downloaded a video but how can I play it? -Once the video is fully downloaded, use any video player, such as vlc or -mplayer. +Once the video is fully downloaded, use any video player, such as mpv, +vlc or mplayer. + +I extracted a video URL with -g, but it does not play on another machine / in my web browser. + +It depends a lot on the service. In many cases, requests for the video +(to download/play it) must come from the same IP address and with the +same cookies and/or HTTP headers. Use the --cookies option to write the +required cookies into a file, and advise your downloader to read cookies +from that file. Some sites also require a common user agent to be used, +use --dump-user-agent to see the one in use by youtube-dl. You can also +get necessary cookies and HTTP headers from JSON output obtained with +--dump-json. + +It may be beneficial to use IPv6; in some cases, the restrictions are +only applied to IPv4. Some services (sometimes only for a subset of +videos) do not restrict the video URL by IP address, cookie, or +user-agent, but these are the exception rather than the rule. -The links provided by youtube-dl -g are not working anymore +Please bear in mind that some URL protocols are NOT supported by +browsers out of the box, including RTMP. If you are using -g, your own +downloader must support these as well. -The URLs youtube-dl outputs require the downloader to have the correct -cookies. Use the --cookies option to write the required cookies into a -file, and advise your downloader to read cookies from that file. Some -sites also require a common user agent to be used, use --dump-user-agent -to see the one in use by youtube-dl. +If you want to play the video on a machine that is not running +youtube-dl, you can relay the video content from the machine that runs +youtube-dl. You can use -o - to let youtube-dl stream a video to stdout, +or simply allow the player to download the files written by youtube-dl +in turn. ERROR: no fmt_url_map or conn information found in video info -youtube has switched to a new video info format in July 2011 which is -not supported by old versions of youtube-dl. You can update youtube-dl -with sudo youtube-dl --update. +YouTube has switched to a new video info format in July 2011 which is +not supported by old versions of youtube-dl. See above for how to update +youtube-dl. ERROR: unable to download video -youtube requires an additional signature since September 2012 which is -not supported by old versions of youtube-dl. You can update youtube-dl -with sudo youtube-dl --update. +YouTube requires an additional signature since September 2012 which is +not supported by old versions of youtube-dl. See above for how to update +youtube-dl. + +Video URL contains an ampersand and I'm getting some strange output [1] 2839 or 'v' is not recognized as an internal or external command + +That's actually the output from your shell. Since ampersand is one of +the special shell characters it's interpreted by the shell preventing +you from passing the whole URL to youtube-dl. To disable your shell from +interpreting the ampersands (or any other special characters) you have +to either put the whole URL in quotes or escape them with a backslash +(which approach will work depends on your shell). + +For example if your URL is +https://www.youtube.com/watch?t=4&v=BaW_jenozKc you should end up with +following command: + +youtube-dl 'https://www.youtube.com/watch?t=4&v=BaW_jenozKc' + +or + +youtube-dl https://www.youtube.com/watch?t=4\&v=BaW_jenozKc + +For Windows you have to use the double quotes: + +youtube-dl "https://www.youtube.com/watch?t=4&v=BaW_jenozKc" + +ExtractorError: Could not find JS function u'OF' + +In February 2015, the new YouTube player contained a character sequence +in a string that was misinterpreted by old versions of youtube-dl. See +above for how to update youtube-dl. + +HTTP Error 429: Too Many Requests or 402: Payment Required + +These two error codes indicate that the service is blocking your IP +address because of overuse. Contact the service and ask them to unblock +your IP address, or - if you have acquired a whitelisted IP address +already - use the --proxy or --source-address options to select another +IP address. SyntaxError: Non-ASCII character @@ -336,37 +1124,843 @@ systems) or clone the git repository, as laid out above. If you modify the code, you can run it by executing the __main__.py file. To recompile the executable, run make youtube-dl. -The exe throws a Runtime error from Visual C++ +The exe throws an error due to missing MSVCR100.dll -To run the exe you need to install first the Microsoft Visual C++ 2008 -Redistributable Package. +To run the exe you need to install first the Microsoft Visual C++ 2010 +Redistributable Package (x86). -COPYRIGHT -========= +On Windows, how should I set up ffmpeg and youtube-dl? Where should I put the exe files? + +If you put youtube-dl and ffmpeg in the same directory that you're +running the command from, it will work, but that's rather cumbersome. + +To make a different directory work - either for ffmpeg, or for +youtube-dl, or for both - simply create the directory (say, C:\bin, or +C:\Users\<User name>\bin), put all the executables directly in there, +and then set your PATH environment variable to include that directory. + +From then on, after restarting your shell, you will be able to access +both youtube-dl and ffmpeg (and youtube-dl will be able to find ffmpeg) +by simply typing youtube-dl or ffmpeg, no matter what directory you're +in. + +How do I put downloads into a specific folder? + +Use the -o to specify an output template, for example +-o "/home/user/videos/%(title)s-%(id)s.%(ext)s". If you want this for +all of your downloads, put the option into your configuration file. + +How do I download a video starting with a -? + +Either prepend https://www.youtube.com/watch?v= or separate the ID from +the options with --: + + youtube-dl -- -wNyEUrxzFU + youtube-dl "https://www.youtube.com/watch?v=-wNyEUrxzFU" + +How do I pass cookies to youtube-dl? + +Use the --cookies option, for example +--cookies /path/to/cookies/file.txt. + +In order to extract cookies from browser use any conforming browser +extension for exporting cookies. For example, cookies.txt (for Chrome) +or cookies.txt (for Firefox). + +Note that the cookies file must be in Mozilla/Netscape format and the +first line of the cookies file must be either # HTTP Cookie File or +# Netscape HTTP Cookie File. Make sure you have correct newline format +in the cookies file and convert newlines if necessary to correspond with +your OS, namely CRLF (\r\n) for Windows and LF (\n) for Unix and +Unix-like systems (Linux, macOS, etc.). HTTP Error 400: Bad Request when +using --cookies is a good sign of invalid newline format. + +Passing cookies to youtube-dl is a good way to workaround login when a +particular extractor does not implement it explicitly. Another use case +is working around CAPTCHA some websites require you to solve in +particular cases in order to get access (e.g. YouTube, CloudFlare). + +How do I stream directly to media player? + +You will first need to tell youtube-dl to stream media to stdout with +-o -, and also tell your media player to read from stdin (it must be +capable of this for streaming) and then pipe former to latter. For +example, streaming to vlc can be achieved with: + + youtube-dl -o - "https://www.youtube.com/watch?v=BaW_jenozKcj" | vlc - + +How do I download only new videos from a playlist? + +Use download-archive feature. With this feature you should initially +download the complete playlist with +--download-archive /path/to/download/archive/file.txt that will record +identifiers of all the videos in a special file. Each subsequent run +with the same --download-archive will download only new videos and skip +all videos that have been downloaded before. Note that only successful +downloads are recorded in the file. + +For example, at first, + + youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re" + +will download the complete PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re playlist +and create a file archive.txt. Each subsequent run will only download +new videos if any: + + youtube-dl --download-archive archive.txt "https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re" + +Should I add --hls-prefer-native into my config? + +When youtube-dl detects an HLS video, it can download it either with the +built-in downloader or ffmpeg. Since many HLS streams are slightly +invalid and ffmpeg/youtube-dl each handle some invalid cases better than +the other, there is an option to switch the downloader if needed. + +When youtube-dl knows that one particular downloader works better for a +given website, that downloader will be picked. Otherwise, youtube-dl +will pick the best downloader for general compatibility, which at the +moment happens to be ffmpeg. This choice may change in future versions +of youtube-dl, with improvements of the built-in downloader and/or +ffmpeg. + +In particular, the generic extractor (used when your website is not in +the list of supported sites by youtube-dl cannot mandate one specific +downloader. + +If you put either --hls-prefer-native or --hls-prefer-ffmpeg into your +configuration, a different subset of videos will fail to download +correctly. Instead, it is much better to file an issue or a pull request +which details why the native or the ffmpeg HLS downloader is a better +choice for your use case. + +Can you add support for this anime video site, or site which shows current movies for free? + +As a matter of policy (as well as legality), youtube-dl does not include +support for services that specialize in infringing copyright. As a rule +of thumb, if you cannot easily find a video that the service is quite +obviously allowed to distribute (i.e. that has been uploaded by the +creator, the creator's distributor, or is published under a free +license), the service is probably unfit for inclusion to youtube-dl. + +A note on the service that they don't host the infringing content, but +just link to those who do, is evidence that the service should NOT be +included into youtube-dl. The same goes for any DMCA note when the whole +front page of the service is filled with videos they are not allowed to +distribute. A "fair use" note is equally unconvincing if the service +shows copyright-protected videos in full without authorization. + +Support requests for services that DO purchase the rights to distribute +their content are perfectly fine though. If in doubt, you can simply +include a source that mentions the legitimate purchase of content. + +How can I speed up work on my issue? + +(Also known as: Help, my important issue not being solved!) The +youtube-dl core developer team is quite small. While we do our best to +solve as many issues as possible, sometimes that can take quite a while. +To speed up your issue, here's what you can do: + +First of all, please do report the issue at our issue tracker. That +allows us to coordinate all efforts by users and developers, and serves +as a unified point. Unfortunately, the youtube-dl project has grown too +large to use personal email as an effective communication channel. + +Please read the bug reporting instructions below. A lot of bugs lack all +the necessary information. If you can, offer proxy, VPN, or shell access +to the youtube-dl developers. If you are able to, test the issue from +multiple computers in multiple countries to exclude local censorship or +misconfiguration issues. + +If nobody is interested in solving your issue, you are welcome to take +matters into your own hands and submit a pull request (or coerce/pay +somebody else to do so). + +Feel free to bump the issue from time to time by writing a small comment +("Issue is still present in youtube-dl version ...from France, but fixed +from Belgium"), but please not more than once a month. Please do not +declare your issue as important or urgent. + +How can I detect whether a given URL is supported by youtube-dl? + +For one, have a look at the list of supported sites. Note that it can +sometimes happen that the site changes its URL scheme (say, from +https://example.com/video/1234567 to https://example.com/v/1234567 ) and +youtube-dl reports an URL of a service in that list as unsupported. In +that case, simply report a bug. + +It is _not_ possible to detect whether a URL is supported or not. That's +because youtube-dl contains a generic extractor which matches ALL URLs. +You may be tempted to disable, exclude, or remove the generic extractor, +but the generic extractor not only allows users to extract videos from +lots of websites that embed a video from another service, but may also +be used to extract video from a service that it's hosting itself. +Therefore, we neither recommend nor support disabling, excluding, or +removing the generic extractor. + +If you want to find out whether a given URL is supported, simply call +youtube-dl with it. If you get no videos back, chances are the URL is +either not referring to a video or unsupported. You can find out which +by examining the output (if you run youtube-dl on the console) or +catching an UnsupportedError exception if you run it from a Python +program. + + + +WHY DO I NEED TO GO THROUGH THAT MUCH RED TAPE WHEN FILING BUGS? + + +Before we had the issue template, despite our extensive bug reporting +instructions, about 80% of the issue reports we got were useless, for +instance because people used ancient versions hundreds of releases old, +because of simple syntactic errors (not in youtube-dl but in general +shell usage), because the problem was already reported multiple times +before, because people did not actually read an error message, even if +it said "please install ffmpeg", because people did not mention the URL +they were trying to download and many more simple, easy-to-avoid +problems, many of whom were totally unrelated to youtube-dl. + +youtube-dl is an open-source project manned by too few volunteers, so +we'd rather spend time fixing bugs where we are certain none of those +simple problems apply, and where we can be reasonably confident to be +able to reproduce the issue without asking the reporter repeatedly. As +such, the output of youtube-dl -v YOUR_URL_HERE is really all that's +required to file an issue. The issue template also guides you through +some basic steps you can do, such as checking that your version of +youtube-dl is current. + + + +DEVELOPER INSTRUCTIONS + + +Most users do not need to build youtube-dl and can download the builds +or get them from their distribution. + +To run youtube-dl as a developer, you don't need to build anything +either. Simply execute + + python -m youtube_dl + +To run the test, simply invoke your favorite test runner, or execute a +test file directly; any of the following work: + + python -m unittest discover + python test/test_download.py + nosetests + +See item 6 of new extractor tutorial for how to run extractor specific +test cases. + +If you want to create a build of youtube-dl yourself, you'll need + +- python +- make (only GNU make is supported) +- pandoc +- zip +- nosetests + +Adding support for a new site + +If you want to add support for a new site, first of all MAKE SURE this +site is NOT DEDICATED TO COPYRIGHT INFRINGEMENT. youtube-dl does NOT +SUPPORT such sites thus pull requests adding support for them WILL BE +REJECTED. + +After you have ensured this site is distributing its content legally, +you can follow this quick list (assuming your service is called +yourextractor): + +1. Fork this repository +2. Check out the source code with: + + git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git + +3. Start a new git branch with + + cd youtube-dl + git checkout -b yourextractor + +4. Start with this simple template and save it to + youtube_dl/extractor/yourextractor.py: + + # coding: utf-8 + from __future__ import unicode_literals + + from .common import InfoExtractor + + + class YourExtractorIE(InfoExtractor): + _VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)' + _TEST = { + 'url': 'https://yourextractor.com/watch/42', + 'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)', + 'info_dict': { + 'id': '42', + 'ext': 'mp4', + 'title': 'Video title goes here', + 'thumbnail': r're:^https?://.*\.jpg$', + # TODO more properties, either as: + # * A value + # * MD5 checksum; start the string with md5: + # * A regular expression; start the string with re: + # * Any Python type (for example int or float) + } + } + + def _real_extract(self, url): + video_id = self._match_id(url) + webpage = self._download_webpage(url, video_id) + + # TODO more code goes here, for example ... + title = self._html_search_regex(r'<h1>(.+?)</h1>', webpage, 'title') + + return { + 'id': video_id, + 'title': title, + 'description': self._og_search_description(webpage), + 'uploader': self._search_regex(r'<div[^>]+id="uploader"[^>]*>([^<]+)<', webpage, 'uploader', fatal=False), + # TODO more properties (see youtube_dl/extractor/common.py) + } + +5. Add an import in youtube_dl/extractor/extractors.py. +6. Run python test/test_download.py TestDownload.test_YourExtractor. + This _should fail_ at first, but you can continually re-run it until + you're done. If you decide to add more than one test, then rename + _TEST to _TESTS and make it into a list of dictionaries. The tests + will then be named TestDownload.test_YourExtractor, + TestDownload.test_YourExtractor_1, + TestDownload.test_YourExtractor_2, etc. Note that tests with + only_matching key in test's dict are not counted in. +7. Have a look at youtube_dl/extractor/common.py for possible helper + methods and a detailed description of what your extractor should and + may return. Add tests and code for as many as you want. +8. Make sure your code follows youtube-dl coding conventions and check + the code with flake8: + + $ flake8 youtube_dl/extractor/yourextractor.py + +9. Make sure your code works under all Python versions claimed + supported by youtube-dl, namely 2.6, 2.7, and 3.2+. +10. When the tests pass, add the new files and commit them and push the + result, like this: + + $ git add youtube_dl/extractor/extractors.py + $ git add youtube_dl/extractor/yourextractor.py + $ git commit -m '[yourextractor] Add new extractor' + $ git push origin yourextractor + +11. Finally, create a pull request. We'll then review and merge it. + +In any case, thank you very much for your contributions! + + +youtube-dl coding conventions + +This section introduces a guide lines for writing idiomatic, robust and +future-proof extractor code. + +Extractors are very fragile by nature since they depend on the layout of +the source data provided by 3rd party media hosters out of your control +and this layout tends to change. As an extractor implementer your task +is not only to write code that will extract media links and metadata +correctly but also to minimize dependency on the source's layout and +even to make the code foresee potential future changes and be ready for +that. This is important because it will allow the extractor not to break +on minor layout changes thus keeping old youtube-dl versions working. +Even though this breakage issue is easily fixed by emitting a new +version of youtube-dl with a fix incorporated, all the previous versions +become broken in all repositories and distros' packages that may not be +so prompt in fetching the update from us. Needless to say, some non +rolling release distros may never receive an update at all. + +Mandatory and optional metafields + +For extraction to work youtube-dl relies on metadata your extractor +extracts and provides to youtube-dl expressed by an information +dictionary or simply _info dict_. Only the following meta fields in the +_info dict_ are considered mandatory for a successful extraction process +by youtube-dl: + +- id (media identifier) +- title (media title) +- url (media download URL) or formats + +In fact only the last option is technically mandatory (i.e. if you can't +figure out the download location of the media the extraction does not +make any sense). But by convention youtube-dl also treats id and title +as mandatory. Thus the aforementioned metafields are the critical data +that the extraction does not make any sense without and if any of them +fail to be extracted then the extractor is considered completely broken. + +Any field apart from the aforementioned ones are considered OPTIONAL. +That means that extraction should be TOLERANT to situations when sources +for these fields can potentially be unavailable (even if they are always +available at the moment) and FUTURE-PROOF in order not to break the +extraction of general purpose mandatory fields. + +Example + +Say you have some source dictionary meta that you've fetched as JSON +with HTTP request and it has a key summary: + + meta = self._download_json(url, video_id) + +Assume at this point meta's layout is: + + { + ... + "summary": "some fancy summary text", + ... + } + +Assume you want to extract summary and put it into the resulting info +dict as description. Since description is an optional meta field you +should be ready that this key may be missing from the meta dict, so that +you should extract it like: + + description = meta.get('summary') # correct + +and not like: + + description = meta['summary'] # incorrect + +The latter will break extraction process with KeyError if summary +disappears from meta at some later time but with the former approach +extraction will just go ahead with description set to None which is +perfectly fine (remember None is equivalent to the absence of data). + +Similarly, you should pass fatal=False when extracting optional data +from a webpage with _search_regex, _html_search_regex or similar +methods, for instance: + + description = self._search_regex( + r'<span[^>]+id="title"[^>]*>([^<]+)<', + webpage, 'description', fatal=False) + +With fatal set to False if _search_regex fails to extract description it +will emit a warning and continue extraction. + +You can also pass default=<some fallback value>, for example: + + description = self._search_regex( + r'<span[^>]+id="title"[^>]*>([^<]+)<', + webpage, 'description', default=None) + +On failure this code will silently continue the extraction with +description set to None. That is useful for metafields that may or may +not be present. + +Provide fallbacks + +When extracting metadata try to do so from multiple sources. For example +if title is present in several places, try extracting from at least some +of them. This makes it more future-proof in case some of the sources +become unavailable. + +Example + +Say meta from the previous example has a title and you are about to +extract it. Since title is a mandatory meta field you should end up with +something like: + + title = meta['title'] + +If title disappears from meta in future due to some changes on the +hoster's side the extraction would fail since title is mandatory. That's +expected. + +Assume that you have some another source you can extract title from, for +example og:title HTML meta of a webpage. In this case you can provide a +fallback scenario: + + title = meta.get('title') or self._og_search_title(webpage) + +This code will try to extract from meta first and if it fails it will +try extracting og:title from a webpage. + +Regular expressions + +Don't capture groups you don't use + +Capturing group must be an indication that it's used somewhere in the +code. Any group that is not used must be non capturing. + +Example + +Don't capture id attribute name here since you can't use it for anything +anyway. + +Correct: + + r'(?:id|ID)=(?P<id>\d+)' + +Incorrect: + + r'(id|ID)=(?P<id>\d+)' + +Make regular expressions relaxed and flexible + +When using regular expressions try to write them fuzzy, relaxed and +flexible, skipping insignificant parts that are more likely to change, +allowing both single and double quotes for quoted values and so on. + +Example + +Say you need to extract title from the following HTML code: + + <span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">some fancy title</span> + +The code for that task should look similar to: + + title = self._search_regex( + r'<span[^>]+class="title"[^>]*>([^<]+)', webpage, 'title') + +Or even better: + + title = self._search_regex( + r'<span[^>]+class=(["\'])title\1[^>]*>(?P<title>[^<]+)', + webpage, 'title', group='title') + +Note how you tolerate potential changes in the style attribute's value +or switch from using double quotes to single for class attribute: + +The code definitely should not look like: + + title = self._search_regex( + r'<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">(.*?)</span>', + webpage, 'title', group='title') + +Long lines policy + +There is a soft limit to keep lines of code under 80 characters long. +This means it should be respected if possible and if it does not make +readability and code maintenance worse. + +For example, you should NEVER split long string literals like URLs or +some other often copied entities over multiple lines to fit this limit: + +Correct: + + 'https://www.youtube.com/watch?v=FqZTN594JQw&list=PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4' + +Incorrect: + + 'https://www.youtube.com/watch?v=FqZTN594JQw&list=' + 'PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4' + +Inline values + +Extracting variables is acceptable for reducing code duplication and +improving readability of complex expressions. However, you should avoid +extracting variables used only once and moving them to opposite parts of +the extractor file, which makes reading the linear flow difficult. + +Example + +Correct: + + title = self._html_search_regex(r'<title>([^<]+)', webpage, 'title') + +Incorrect: + + TITLE_RE = r'([^<]+)' + # ...some lines of code... + title = self._html_search_regex(TITLE_RE, webpage, 'title') + +Collapse fallbacks + +Multiple fallback values can quickly become unwieldy. Collapse multiple +fallback values into a single expression via a list of patterns. + +Example + +Good: + + description = self._html_search_meta( + ['og:description', 'description', 'twitter:description'], + webpage, 'description', default=None) + +Unwieldy: + + description = ( + self._og_search_description(webpage, default=None) + or self._html_search_meta('description', webpage, default=None) + or self._html_search_meta('twitter:description', webpage, default=None)) + +Methods supporting list of patterns are: _search_regex, +_html_search_regex, _og_search_property, _html_search_meta. + +Trailing parentheses + +Always move trailing parentheses after the last argument. + +Example + +Correct: + + lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], + list) + +Incorrect: + + lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], + list, + ) + +Use convenience conversion and parsing functions + +Wrap all extracted numeric data into safe functions from +youtube_dl/utils.py: int_or_none, float_or_none. Use them for string to +number conversions as well. + +Use url_or_none for safe URL processing. + +Use try_get for safe metadata extraction from parsed JSON. + +Use unified_strdate for uniform upload_date or any YYYYMMDD meta field +extraction, unified_timestamp for uniform timestamp extraction, +parse_filesize for filesize extraction, parse_count for count meta +fields extraction, parse_resolution, parse_duration for duration +extraction, parse_age_limit for age_limit extraction. + +Explore youtube_dl/utils.py for more useful convenience functions. + +More examples + +Safely extract optional description from parsed JSON + + description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str) + +Safely extract more optional metadata + + video = try_get(response, lambda x: x['result']['video'][0], dict) or {} + description = video.get('summary') + duration = float_or_none(video.get('durationMs'), scale=1000) + view_count = int_or_none(video.get('views')) + + + +EMBEDDING YOUTUBE-DL + + +youtube-dl makes the best effort to be a good command-line program, and +thus should be callable from any programming language. If you encounter +any problems parsing its output, feel free to create a report. + +From a Python program, you can embed youtube-dl in a more powerful +fashion, like this: + + from __future__ import unicode_literals + import youtube_dl + + ydl_opts = {} + with youtube_dl.YoutubeDL(ydl_opts) as ydl: + ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc']) + +Most likely, you'll want to use various options. For a list of options +available, have a look at youtube_dl/YoutubeDL.py. For a start, if you +want to intercept youtube-dl's output, set a logger object. + +Here's a more complete example of a program that outputs only errors +(and a short message after the download is finished), and +downloads/converts the video to an mp3 file: + + from __future__ import unicode_literals + import youtube_dl + + + class MyLogger(object): + def debug(self, msg): + pass + + def warning(self, msg): + pass + + def error(self, msg): + print(msg) + + + def my_hook(d): + if d['status'] == 'finished': + print('Done downloading, now converting ...') + + + ydl_opts = { + 'format': 'bestaudio/best', + 'postprocessors': [{ + 'key': 'FFmpegExtractAudio', + 'preferredcodec': 'mp3', + 'preferredquality': '192', + }], + 'logger': MyLogger(), + 'progress_hooks': [my_hook], + } + with youtube_dl.YoutubeDL(ydl_opts) as ydl: + ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc']) -youtube-dl is released into the public domain by the copyright holders. -This README file was originally written by Daniel Bolton -(https://github.com/dbbolton) and is likewise released into the public -domain. BUGS -==== + Bugs and suggestions should be reported at: -https://github.com/rg3/youtube-dl/issues - -Please include: - -- Your exact command line, like - youtube-dl -t "http://www.youtube.com/watch?v=uHlDtZ6Oc3s&feature=channel_video_title". - A common mistake is not to escape the &. Putting URLs in quotes - should solve this problem. -- If possible re-run the command with --verbose, and include the full - output, it is really helpful to us. -- The output of youtube-dl --version -- The output of python --version -- The name and version of your Operating System ("Ubuntu 11.04 x64" or - "Windows 7 x64" is usually enough). - -For discussions, join us in the irc channel #youtube-dl on freenode. +https://github.com/ytdl-org/youtube-dl/issues. Unless you were prompted +to or there is another pertinent reason (e.g. GitHub fails to accept the +bug report), please do not send bug reports via personal email. For +discussions, join us in the IRC channel #youtube-dl on freenode +(webchat). + +PLEASE INCLUDE THE FULL OUTPUT OF YOUTUBE-DL WHEN RUN WITH -v, i.e. ADD +-v flag to YOUR COMMAND LINE, copy the WHOLE output and post it in the +issue body wrapped in ``` for better formatting. It should look similar +to this: + + $ youtube-dl -v + [debug] System config: [] + [debug] User config: [] + [debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKcj'] + [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251 + [debug] youtube-dl version 2015.12.06 + [debug] Git HEAD: 135392e + [debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2 + [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4 + [debug] Proxy map: {} + ... + +DO NOT POST SCREENSHOTS OF VERBOSE LOGS; ONLY PLAIN TEXT IS ACCEPTABLE. + +The output (including the first lines) contains important debugging +information. Issues without the full output are often not reproducible +and therefore do not get solved in short order, if ever. + +Please re-read your issue once again to avoid a couple of common +mistakes (you can and should use this as a checklist): + +Is the description of the issue itself sufficient? + +We often get issue reports that we cannot really decipher. While in most +cases we eventually get the required information after asking back +multiple times, this poses an unnecessary drain on our resources. Many +contributors, including myself, are also not native speakers, so we may +misread some parts. + +So please elaborate on what feature you are requesting, or what bug you +want to be fixed. Make sure that it's obvious + +- What the problem is +- How it could be fixed +- How your proposed solution would look like + +If your report is shorter than two lines, it is almost certainly missing +some of these, which makes it hard for us to respond to it. We're often +too polite to close the issue outright, but the missing info makes +misinterpretation likely. As a committer myself, I often get frustrated +by these issues, since the only possible way for me to move forward on +them is to ask for clarification over and over. + +For bug reports, this means that your report should contain the +_complete_ output of youtube-dl when called with the -v flag. The error +message you get for (most) bugs even says so, but you would not believe +how many of our bug reports do not contain this information. + +If your server has multiple IPs or you suspect censorship, adding +--call-home may be a good idea to get more diagnostics. If the error is +ERROR: Unable to extract ... and you cannot reproduce it from multiple +countries, add --dump-pages (warning: this will yield a rather large +output, redirect it to the file log.txt by adding >log.txt 2>&1 to your +command-line) or upload the .dump files you get when you add +--write-pages somewhere. + +SITE SUPPORT REQUESTS MUST CONTAIN AN EXAMPLE URL. An example URL is a +URL you might want to download, like +https://www.youtube.com/watch?v=BaW_jenozKc. There should be an obvious +video present. Except under very special circumstances, the main page of +a video service (e.g. https://www.youtube.com/) is _not_ an example URL. + +Are you using the latest version? + +Before reporting any issue, type youtube-dl -U. This should report that +you're up-to-date. About 20% of the reports we receive are already +fixed, but people are using outdated versions. This goes for feature +requests as well. + +Is the issue already documented? + +Make sure that someone has not already opened the issue you're trying to +open. Search at the top of the window or browse the GitHub Issues of +this repository. If there is an issue, feel free to write something +along the lines of "This affects me as well, with version 2015.01.01. +Here is some more information on the issue: ...". While some issues may +be old, a new post into them often spurs rapid activity. + +Why are existing options not enough? + +Before requesting a new feature, please have a quick peek at the list of +supported options. Many feature requests are for features that actually +exist already! Please, absolutely do show off your work in the issue +report and detail how the existing similar options do _not_ solve your +problem. + +Is there enough context in your bug report? + +People want to solve problems, and often think they do us a favor by +breaking down their larger problems (e.g. wanting to skip already +downloaded files) to a specific request (e.g. requesting us to look +whether the file exists before downloading the info page). However, what +often happens is that they break down the problem into two steps: One +simple, and one impossible (or extremely complicated one). + +We are then presented with a very complicated request when the original +problem could be solved far easier, e.g. by recording the downloaded +video IDs in a separate file. To avoid this, you must include the +greater context where it is non-obvious. In particular, every feature +request that does not consist of adding support for a new site should +contain a use case scenario that explains in what situation the missing +feature would be useful. + +Does the issue involve one problem, and one problem only? + +Some of our users seem to think there is a limit of issues they can or +should open. There is no limit of issues they can or should open. While +it may seem appealing to be able to dump all your issues into one +ticket, that means that someone who solves one of your issues cannot +mark the issue as closed. Typically, reporting a bunch of issues leads +to the ticket lingering since nobody wants to attack that behemoth, +until someone mercifully splits the issue into multiple ones. + +In particular, every site support request issue should only pertain to +services at one site (generally under a common domain, but always using +the same backend technology). Do not request support for vimeo user +videos, White house podcasts, and Google Plus pages in the same issue. +Also, make sure that you don't post bug reports alongside feature +requests. As a rule of thumb, a feature request does not include outputs +of youtube-dl that are not immediately related to the feature at hand. +Do not post reports of a network error alongside the request for a new +video service. + +Is anyone going to need the feature? + +Only post features that you (or an incapacitated friend you can +personally talk to) require. Do not post features because they seem like +a good idea. If they are really useful, they will be requested by +someone who requires them. + +Is your question about youtube-dl? + +It may sound strange, but some bug reports we receive are completely +unrelated to youtube-dl and relate to a different, or even the +reporter's own, application. Please make sure that you are actually +using youtube-dl. If you are using a UI for youtube-dl, report the bug +to the maintainer of the actual application providing the UI. On the +other hand, if your UI for youtube-dl fails in some way you believe is +related to youtube-dl, by all means, go ahead and report the bug. + + + +COPYRIGHT + + +youtube-dl is released into the public domain by the copyright holders. + +This README file was originally written by Daniel Bolton and is likewise +released into the public domain.