X-Git-Url: https://git.rapsys.eu/youtubedl/blobdiff_plain/a4409a39ae30315a4f0c7cc8eb7c8f9a937225bc..bad87b6c6e31ea0056d795ca3e7aa81ceaad2291:/README.txt diff --git a/README.txt b/README.txt index 24959f0..974ea6e 100644 --- a/README.txt +++ b/README.txt @@ -20,7 +20,7 @@ youtube-dl - download videos from youtube.com or other video platforms INSTALLATION -To install it right away for all UNIX users (Linux, OS X, etc.), type: +To install it right away for all UNIX users (Linux, macOS, etc.), type: sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl sudo chmod a+rx /usr/local/bin/youtube-dl @@ -41,7 +41,7 @@ You can also use pip: This command will update youtube-dl if you have already installed it. See the pypi page for more information. -OS X users can install youtube-dl with Homebrew: +macOS users can install youtube-dl with Homebrew: brew install youtube-dl @@ -116,8 +116,8 @@ OPTIONS Network Options: --proxy URL Use the specified HTTP/HTTPS/SOCKS proxy. - To enable experimental SOCKS proxy, specify - a proper scheme. For example + To enable SOCKS proxy, specify a proper + scheme. For example socks5://127.0.0.1:1080/. Pass in an empty string (--proxy "") for direct connection --socket-timeout SECONDS Time to wait before giving up, in seconds @@ -131,16 +131,18 @@ Geo Restriction: --geo-verification-proxy URL Use this proxy to verify the IP address for some geo-restricted sites. The default proxy specified by --proxy (or none, if the - options is not present) is used for the + option is not present) is used for the actual downloading. --geo-bypass Bypass geographic restriction via faking - X-Forwarded-For HTTP header (experimental) + X-Forwarded-For HTTP header --no-geo-bypass Do not bypass geographic restriction via faking X-Forwarded-For HTTP header - (experimental) --geo-bypass-country CODE Force bypass geographic restriction with explicitly provided two-letter ISO 3166-2 - country code (experimental) + country code + --geo-bypass-ip-block IP_BLOCK Force bypass geographic restriction with + explicitly provided IP block in CIDR + notation Video Selection: @@ -235,7 +237,7 @@ Download Options: --playlist-reverse Download playlist videos in reverse order --playlist-random Download playlist videos in random order --xattr-set-filesize Set file xattribute ytdl.filesize with - expected file size (experimental) + expected file size --hls-prefer-native Use the native HLS downloader instead of ffmpeg --hls-prefer-ffmpeg Use ffmpeg instead of the native HLS @@ -254,7 +256,9 @@ Download Options: Filesystem Options: -a, --batch-file FILE File containing URLs to download ('-' for - stdin) + stdin), one URL per line. Lines starting + with '#', ';' or ']' are considered as + comments and ignored. --id Use only video ID in file name -o, --output TEMPLATE Output filename template, see the "OUTPUT TEMPLATE" for all the info @@ -470,16 +474,16 @@ Post-processing Options: default; fix file if we can, warn otherwise) --prefer-avconv Prefer avconv over ffmpeg for running the - postprocessors (default) - --prefer-ffmpeg Prefer ffmpeg over avconv for running the postprocessors + --prefer-ffmpeg Prefer ffmpeg over avconv for running the + postprocessors (default) --ffmpeg-location PATH Location of the ffmpeg/avconv binary; either the path to the binary or its containing directory. --exec CMD Execute a command on the file after - downloading, similar to find's -exec - syntax. Example: --exec 'adb push {} - /sdcard/Music/ && rm {}' + downloading and post-processing, similar to + find's -exec syntax. Example: --exec 'adb + push {} /sdcard/Music/ && rm {}' --convert-subs FORMAT Convert the subtitles to other format (currently supported: srt|ass|vtt|lrc) @@ -489,7 +493,7 @@ CONFIGURATION You can configure youtube-dl by placing any supported command line -option to a configuration file. On Linux and OS X, the system wide +option to a configuration file. On Linux and macOS, the system wide configuration file is located at /etc/youtube-dl.conf and the user wide configuration file at ~/.config/youtube-dl/config. On Windows, the user wide configuration file locations are %APPDATA%\youtube-dl\config.txt or @@ -572,8 +576,8 @@ However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to python string formatting operations. For example, %(NAME)s or %(NAME)05d. To clarify, that is a percent symbol followed by a name in -parentheses, followed by a formatting operations. Allowed names along -with sequence type are: +parentheses, followed by formatting operations. Allowed names along with +sequence type are: - id (string): Video identifier - title (string): Video title @@ -590,6 +594,8 @@ with sequence type are: available - upload_date (string): Video upload date (YYYYMMDD) - uploader_id (string): Nickname or id of the video uploader +- channel (string): Full name of the channel the video is uploaded on +- channel_id (string): Id of the channel - location (string): Physical location where the video was filmed - duration (numeric): Length of the video in seconds - view_count (numeric): How many users have watched the video on the @@ -773,15 +779,20 @@ of a particular file extension served as a single file, e.g. -f webm will download the best quality format with the webm extension served as a single file. -You can also use special names to select particular edge case formats: - -best: Select the best quality format represented by a single file with -video and audio. - worst: Select the worst quality format represented by -a single file with video and audio. - bestvideo: Select the best quality -video-only format (e.g. DASH video). May not be available. - worstvideo: -Select the worst quality video-only format. May not be available. - -bestaudio: Select the best quality audio only-format. May not be -available. - worstaudio: Select the worst quality audio only-format. May -not be available. +You can also use special names to select particular edge case formats: + +- best: Select the best quality format represented by a single file + with video and audio. +- worst: Select the worst quality format represented by a single file + with video and audio. +- bestvideo: Select the best quality video-only format (e.g. DASH + video). May not be available. +- worstvideo: Select the worst quality video-only format. May not be + available. +- bestaudio: Select the best quality audio only-format. May not be + available. +- worstaudio: Select the worst quality audio only-format. May not be + available. For example, to download the worst quality video-only format you can use -f worstvideo. @@ -804,20 +815,31 @@ You can also filter the video formats by putting a condition in brackets, as in -f "best[height=720]" (or -f "[filesize>10M]"). The following numeric meta fields can be used with comparisons <, <=, >, ->=, = (equals), != (not equals): - filesize: The number of bytes, if -known in advance - width: Width of the video, if known - height: Height -of the video, if known - tbr: Average bitrate of audio and video in -KBit/s - abr: Average audio bitrate in KBit/s - vbr: Average video -bitrate in KBit/s - asr: Audio sampling rate in Hertz - fps: Frame rate - -Also filtering work for comparisons = (equals), != (not equals), ^= -(begins with), $= (ends with), *= (contains) and following string meta -fields: - ext: File extension - acodec: Name of the audio codec in use - -vcodec: Name of the video codec in use - container: Name of the -container format - protocol: The protocol that will be used for the -actual download, lower-case (http, https, rtsp, rtmp, rtmpe, mms, f4m, -ism, http_dash_segments, m3u8, or m3u8_native) - format_id: A short -description of the format +>=, = (equals), != (not equals): + +- filesize: The number of bytes, if known in advance +- width: Width of the video, if known +- height: Height of the video, if known +- tbr: Average bitrate of audio and video in KBit/s +- abr: Average audio bitrate in KBit/s +- vbr: Average video bitrate in KBit/s +- asr: Audio sampling rate in Hertz +- fps: Frame rate + +Also filtering work for comparisons = (equals), ^= (starts with), $= +(ends with), *= (contains) and following string meta fields: + +- ext: File extension +- acodec: Name of the audio codec in use +- vcodec: Name of the video codec in use +- container: Name of the container format +- protocol: The protocol that will be used for the actual download, + lower-case (http, https, rtsp, rtmp, rtmpe, mms, f4m, ism, + http_dash_segments, m3u8, or m3u8_native) +- format_id: A short description of the format + +Any string comparison may be prefixed with negation ! in order to +produce an opposite comparison, e.g. !*= (does not contain). Note that none of the aforementioned meta fields are guaranteed to be present since this solely depends on the metadata obtained by particular @@ -870,7 +892,7 @@ single. # Download best mp4 format available or any other best if no mp4 available $ youtube-dl -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best' - # Download best format available but not better that 480p + # Download best format available but no better than 480p $ youtube-dl -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' # Download best video only format but no bigger than 50 MB @@ -938,8 +960,8 @@ that, remove the distribution's package, with a line like Afterwards, simply follow our manual installation instructions: - sudo wget https://yt-dl.org/latest/youtube-dl -O /usr/local/bin/youtube-dl - sudo chmod a+x /usr/local/bin/youtube-dl + sudo wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl + sudo chmod a+rx /usr/local/bin/youtube-dl hash -r Again, from then on you'll be able to update with sudo youtube-dl -U. @@ -1079,10 +1101,19 @@ above for how to update youtube-dl. HTTP Error 429: Too Many Requests or 402: Payment Required These two error codes indicate that the service is blocking your IP -address because of overuse. Contact the service and ask them to unblock -your IP address, or - if you have acquired a whitelisted IP address -already - use the --proxy or --source-address options to select another -IP address. +address because of overuse. Usually this is a soft block meaning that +you can gain access again after solving CAPTCHA. Just open a browser and +solve a CAPTCHA the service suggests you and after that pass cookies to +youtube-dl. Note that if your machine has multiple external IPs then you +should also pass exactly the same IP you've used for solving CAPTCHA +with --source-address. Also you may need to pass a User-Agent HTTP +header of your browser with --user-agent. + +If this is not the case (no CAPTCHA suggested to solve by the service) +then you can contact the service and ask them to unblock your IP +address, or - if you have acquired a whitelisted IP address already - +use the --proxy or --source-address options to select another IP +address. SyntaxError: Non-ASCII character @@ -1143,7 +1174,7 @@ Use the --cookies option, for example In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, cookies.txt (for Chrome) -or Export Cookies (for Firefox). +or cookies.txt (for Firefox). Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either # HTTP Cookie File or @@ -1351,12 +1382,12 @@ yourextractor): 1. Fork this repository 2. Check out the source code with: - git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git + git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git 3. Start a new git branch with - cd youtube-dl - git checkout -b yourextractor + cd youtube-dl + git checkout -b yourextractor 4. Start with this simple template and save it to youtube_dl/extractor/yourextractor.py: @@ -1413,10 +1444,13 @@ yourextractor): methods and a detailed description of what your extractor should and may return. Add tests and code for as many as you want. 8. Make sure your code follows youtube-dl coding conventions and check - the code with flake8. Also make sure your code works under all - Python versions claimed supported by youtube-dl, namely 2.6, 2.7, - and 3.2+. -9. When the tests pass, add the new files and commit them and push the + the code with flake8: + + $ flake8 youtube_dl/extractor/yourextractor.py + +9. Make sure your code works under all Python versions claimed + supported by youtube-dl, namely 2.6, 2.7, and 3.2+. +10. When the tests pass, add the new files and commit them and push the result, like this: $ git add youtube_dl/extractor/extractors.py @@ -1424,7 +1458,7 @@ yourextractor): $ git commit -m '[yourextractor] Add new extractor' $ git push origin yourextractor -10. Finally, create a pull request. We'll then review and merge it. +11. Finally, create a pull request. We'll then review and merge it. In any case, thank you very much for your contributions! @@ -1553,9 +1587,31 @@ fallback scenario: This code will try to extract from meta first and if it fails it will try extracting og:title from a webpage. -Make regular expressions flexible +Regular expressions + +Don't capture groups you don't use -When using regular expressions try to write them fuzzy and flexible. +Capturing group must be an indication that it's used somewhere in the +code. Any group that is not used must be non capturing. + +Example + +Don't capture id attribute name here since you can't use it for anything +anyway. + +Correct: + + r'(?:id|ID)=(?P\d+)' + +Incorrect: + + r'(id|ID)=(?P\d+)' + +Make regular expressions relaxed and flexible + +When using regular expressions try to write them fuzzy, relaxed and +flexible, skipping insignificant parts that are more likely to change, +allowing both single and double quotes for quoted values and so on. Example @@ -1583,11 +1639,113 @@ The code definitely should not look like: r'(.*?)', webpage, 'title', group='title') -Use safe conversion functions +Long lines policy + +There is a soft limit to keep lines of code under 80 characters long. +This means it should be respected if possible and if it does not make +readability and code maintenance worse. + +For example, you should NEVER split long string literals like URLs or +some other often copied entities over multiple lines to fit this limit: + +Correct: + + 'https://www.youtube.com/watch?v=FqZTN594JQw&list=PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4' + +Incorrect: + + 'https://www.youtube.com/watch?v=FqZTN594JQw&list=' + 'PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4' + +Inline values + +Extracting variables is acceptable for reducing code duplication and +improving readability of complex expressions. However, you should avoid +extracting variables used only once and moving them to opposite parts of +the extractor file, which makes reading the linear flow difficult. + +Example + +Correct: + + title = self._html_search_regex(r'([^<]+)', webpage, 'title') + +Incorrect: + + TITLE_RE = r'([^<]+)' + # ...some lines of code... + title = self._html_search_regex(TITLE_RE, webpage, 'title') + +Collapse fallbacks + +Multiple fallback values can quickly become unwieldy. Collapse multiple +fallback values into a single expression via a list of patterns. + +Example + +Good: + + description = self._html_search_meta( + ['og:description', 'description', 'twitter:description'], + webpage, 'description', default=None) + +Unwieldy: + + description = ( + self._og_search_description(webpage, default=None) + or self._html_search_meta('description', webpage, default=None) + or self._html_search_meta('twitter:description', webpage, default=None)) + +Methods supporting list of patterns are: _search_regex, +_html_search_regex, _og_search_property, _html_search_meta. + +Trailing parentheses + +Always move trailing parentheses after the last argument. + +Example + +Correct: + + lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], + list) + +Incorrect: + + lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'], + list, + ) + +Use convenience conversion and parsing functions + +Wrap all extracted numeric data into safe functions from +youtube_dl/utils.py: int_or_none, float_or_none. Use them for string to +number conversions as well. + +Use url_or_none for safe URL processing. + +Use try_get for safe metadata extraction from parsed JSON. + +Use unified_strdate for uniform upload_date or any YYYYMMDD meta field +extraction, unified_timestamp for uniform timestamp extraction, +parse_filesize for filesize extraction, parse_count for count meta +fields extraction, parse_resolution, parse_duration for duration +extraction, parse_age_limit for age_limit extraction. + +Explore youtube_dl/utils.py for more useful convenience functions. + +More examples + +Safely extract optional description from parsed JSON + + description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str) + +Safely extract more optional metadata -Wrap all extracted numeric data into safe functions from utils: -int_or_none, float_or_none. Use them for string to number conversions as -well. + video = try_get(response, lambda x: x['result']['video'][0], dict) or {} + description = video.get('summary') + duration = float_or_none(video.get('durationMs'), scale=1000) + view_count = int_or_none(video.get('views')) @@ -1655,9 +1813,9 @@ BUGS Bugs and suggestions should be reported at: -https://github.com/rg3/youtube-dl/issues. Unless you were prompted to or -there is another pertinent reason (e.g. GitHub fails to accept the bug -report), please do not send bug reports via personal email. For +https://github.com/ytdl-org/youtube-dl/issues. Unless you were prompted +to or there is another pertinent reason (e.g. GitHub fails to accept the +bug report), please do not send bug reports via personal email. For discussions, join us in the IRC channel #youtube-dl on freenode (webchat).