fixes

Authored by: bashonly

.github/PULL_REQUEST_TEMPLATE.md

@@ -28,7 +28,7 @@ Fixes #
 ### Before submitting a *pull request* make sure you have:
 - [ ] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
 - [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
-- [ ] Checked the code with [ruff](https://docs.astral.sh/ruff/) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)
+- [ ] Checked the code using `hatch run check` (or ran `hatch run install` before committing) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)
 
 ### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
 - [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)

.github/workflows/quick-test.yml

@@ -20,8 +20,8 @@ jobs:
       run: |
         python3 -m yt_dlp -v || true
         python3 ./devscripts/run_tests.py core
-  linter:
-    name: Linter
+  check:
+    name: Code check
     if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ubuntu-latest
     steps:
@@ -32,4 +32,6 @@ jobs:
     - name: Make lazy extractors
       run: python3 ./devscripts/make_lazy_extractors.py
     - name: Run ruff
-      run: ruff check . --output-format github
+      run: ruff check --output-format github .
+    - name: Run autopep8
+      run: autopep8 --diff .

.gitignore

@@ -64,7 +64,7 @@ cookies
 # Python
 *.pyc
 *.pyo
-.pytest_cache
+.*_cache
 wine-py2exe/
 py2exe.log
 build/

.pre-commit-config.yaml

@@ -1,13 +1,8 @@
 repos:
 - repo: local
   hooks:
-  - id: linter
-    name: linter
-    entry: hatch run lint
-    language: system
-    types: [python]
-  - id: format
-    name: format
-    entry: hatch run format --diff
+  - id: check
+    name: code check
+    entry: hatch run check
     language: system
     types: [python]

CONTRIBUTING.md

@@ -134,35 +134,31 @@ We follow [youtube-dl's policy](https://github.com/ytdl-org/youtube-dl#can-you-a
 
 # DEVELOPER INSTRUCTIONS
 
-Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases) or get them via [the other installation methods](README.md#installation).
+Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases), get them via [the other installation methods](README.md#installation) or directly run it using `python -m yt_dlp`.
 
-To quickly run yt-dlp as a developer, you don't need to build anything either. Simply execute
+`yt-dlp` uses [`hatch`](<https://hatch.pypa.io>) as a project management tool.
+You can easily install it using [`pipx`](<https://pipx.pypa.io>) via `pipx install hatch`, or else via `pip` or your package manager of choice.
 
-```shell
-$ python -m yt_dlp
-```
-
-To properly interact with `yt-dlp` as a developer, you should use [`hatch`](<https://hatch.pypa.io>) (easily installed using [`pipx`](<https://pipx.pypa.io>)).
-After installing, you can use `hatch shell` to create a shell with all dependencies as well as `yt-dlp` installed.
-
-If you plan on contributing to `yt-dlp`, run
+If you plan on contributing to `yt-dlp`, you are required to run
 
 ```shell
 $ hatch run install
 ```
 
-to prepare a full development environment and install `pre-commit`.
+before creating commits so that the pre-commit hooks are properly initialized.
+After this you can use `hatch shell` to enable a virtual environment that has development dependencies as well as `yt-dlp` installed.
 
-To run all the available core tests, use:
+Scripts can be used to run simple tasks, like linting or testing, without having to run `hatch shell` first:
 
 ```shell
-$ hatch run tests core
+$ hatch run lint
+$ hatch run format
+$ hatch run test
 ```
 
-You can also run tests for all installed python versions sequentially by setting the `TEST_ALL` variable, like so:
-
+You can run scripts for all available and supported python versions sequentially by setting the `TEST_ALL` variable:
 ```shell
-$ TEST_ALL=1 hatch run tests core
+$ TEST_ALL=1 hatch run test core
 ```
 
 See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases.
@@ -243,17 +239,16 @@ After you have ensured this site is distributing its content legally, you can fo
             }
     ```
 1. Add an import in [`yt_dlp/extractor/_extractors.py`](yt_dlp/extractor/_extractors.py). Note that the class name must end with `IE`.
-1. Run `hatch run tests YourExtractor`. This *may fail* at first, but you can continually re-run it until you're done. Upon failure, it will output the missing fields and/or correct values which you can copy. If you decide to add more than one test, the tests will then be named `YourExtractor`, `YourExtractor_1`, `YourExtractor_2`, etc. Note that tests with an `only_matching` key in the test's dict are not included in the count. You can also run all the tests in one go with `YourExtractor_all`
+1. Run `hatch run test YourExtractor`. This *may fail* at first, but you can continually re-run it until you're done. Upon failure, it will output the missing fields and/or correct values which you can copy. If you decide to add more than one test, the tests will then be named `YourExtractor`, `YourExtractor_1`, `YourExtractor_2`, etc. Note that tests with an `only_matching` key in the test's dict are not included in the count. You can also run all the tests in one go with `YourExtractor_all`
 1. Make sure you have at least one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the particular test is disabled from running.
 1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L119-L440). Add tests and code for as many as you want.
 1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions), passes [ruff](https://docs.astral.sh/ruff/tutorial/#getting-started) code checks and is properly formatted:
 
     ```shell
-    $ hatch run lint
-    $ hatch run format
+    $ hatch run check
     ```
 
-    You can use `hatch run lint --fix` to automatically fix problems.
+    You can use `hatch run lint` and `hatch run format` to automatically fix problems.
 
 1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.8 and above. Backward compatibility is not required for even older versions of Python.
 1. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files, [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:

Makefile

@@ -10,7 +10,7 @@ tar: yt-dlp.tar.gz
 # intended use: when building a source distribution,
 # make pypi-files && python3 -m build -sn .
 pypi-files: AUTHORS Changelog.md LICENSE README.md README.txt supportedsites \
-	        completions yt-dlp.1 pyproject.toml setup.cfg devscripts/* test/*
+	        completions yt-dlp.1 pyproject.toml devscripts/* test/*
 
 .PHONY: all clean install test tar pypi-files completions ot offlinetest codetest supportedsites
 
@@ -24,7 +24,7 @@ clean-dist:
 	yt_dlp/extractor/lazy_extractors.py *.spec CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe yt_dlp.egg-info/ AUTHORS
 clean-cache:
 	find . \( \
-		-type d -name .pytest_cache -o -type d -name __pycache__ -o -name "*.pyc" -o -name "*.class" \
+		-type d -name '.*_cache' -o -type d -name __pycache__ -o -name "*.pyc" -o -name "*.class" \
 	\) -prune -exec rm -rf {} \;
 
 completion-bash: completions/bash/yt-dlp
@@ -38,11 +38,13 @@ MANDIR ?= $(PREFIX)/man
 SHAREDIR ?= $(PREFIX)/share
 PYTHON ?= /usr/bin/env python3
 
-# set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
-SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi)
+# $(shell) and $(error) are no-ops in BSD Make and the != variable assignment operator is not supported by GNU Make <4.0
+VERSION_CHECK != echo supported
+VERSION_CHECK ?= $(error GNU Make 4+ or BSD Make is required)
+CHECK_VERSION := $(VERSION_CHECK)
 
-# set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
-MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
+# set markdown input format to "markdown-smart" for pandoc version 2+ and to "markdown" for pandoc prior to version 2
+MARKDOWN != if [ "`pandoc -v | head -n1 | cut -d' ' -f2 | head -c1`" -ge "2" ]; then echo markdown-smart; else echo markdown; fi
 
 install: lazy-extractors yt-dlp yt-dlp.1 completions
 	mkdir -p $(DESTDIR)$(BINDIR)
@@ -65,6 +67,7 @@ uninstall:
 
 codetest:
 	ruff check .
+	autopep8 --diff .
 
 test:
 	$(PYTHON) -m pytest
@@ -73,17 +76,17 @@ test:
 offlinetest: codetest
 	$(PYTHON) -m pytest -k "not download"
 
-CODE_FOLDERS := $(shell find yt_dlp -type d -not -name '__*' -exec sh -c 'test -e "$$1"/__init__.py' sh {} \; -print)
-CODE_FILES := $(shell for f in $(CODE_FOLDERS); do echo "$$f" | awk '{gsub(/\/[^\/]+/,"/*"); print $$1"/*.py"}'; done | sort -u)
+CODE_FOLDERS != find yt_dlp -type f -name '__init__.py' -exec dirname {} \+ | grep -v '/__' | sort
+CODE_FILES != for f in $(CODE_FOLDERS) ; do echo "$$f" | sed 's,$$,/*.py,' ; done
 yt-dlp: $(CODE_FILES)
 	mkdir -p zip
 	for d in $(CODE_FOLDERS) ; do \
 	  mkdir -p zip/$$d ;\
 	  cp -pPR $$d/*.py zip/$$d/ ;\
 	done
-	cd zip ; touch -t 200001010101 $(CODE_FILES)
+	(cd zip && touch -t 200001010101 $(CODE_FILES))
 	mv zip/yt_dlp/__main__.py zip/
-	cd zip ; zip -q ../yt-dlp $(CODE_FILES) __main__.py
+	(cd zip && zip -q ../yt-dlp $(CODE_FILES) __main__.py)
 	rm -rf zip
 	echo '#!$(PYTHON)' > yt-dlp
 	cat yt-dlp.zip >> yt-dlp
@@ -127,7 +130,7 @@ completions/fish/yt-dlp.fish: $(CODE_FILES) devscripts/fish-completion.in
 	mkdir -p completions/fish
 	$(PYTHON) devscripts/fish-completion.py
 
-_EXTRACTOR_FILES = $(shell find yt_dlp/extractor -name '*.py' -and -not -name 'lazy_extractors.py')
+_EXTRACTOR_FILES != find yt_dlp/extractor -name '*.py' -and -not -name 'lazy_extractors.py'
 yt_dlp/extractor/lazy_extractors.py: devscripts/make_lazy_extractors.py devscripts/lazy_load_template.py $(_EXTRACTOR_FILES)
 	$(PYTHON) devscripts/make_lazy_extractors.py $@
 
@@ -139,13 +142,14 @@ yt-dlp.tar.gz: all
 		--exclude '*.pyo' \
 		--exclude '*~' \
 		--exclude '__pycache__' \
-		--exclude '.pytest_cache' \
+		--exclude '.*_cache' \
 		--exclude '.git' \
+		--exclude '__pyinstaller' \
 		-- \
 		README.md supportedsites.md Changelog.md LICENSE \
 		CONTRIBUTING.md Collaborators.md CONTRIBUTORS AUTHORS \
 		Makefile yt-dlp.1 README.txt completions .gitignore \
-		setup.cfg yt-dlp yt_dlp pyproject.toml devscripts test
+		yt-dlp yt_dlp pyproject.toml devscripts test
 
 AUTHORS:
 	git shortlog -s -n HEAD | cut -f2 | sort > AUTHORS

devscripts/run_tests.py

@@ -32,9 +32,9 @@ def run_tests(*tests, pattern=None, ci=False):
     if pattern:
         arguments.extend(['-k', pattern])
     if run_core:
-        arguments.extend(['--ignore', 'test/test_download.py'])
+        arguments.extend(['-m', 'not download'])
     elif run_download:
-        arguments.extend(['test/test_download.py'])
+        arguments.extend(['-m', 'download'])
     else:
         arguments.extend(
             f'test/test_download.py::TestDownload::test_{test}' for test in tests)

pyproject.toml

@@ -62,7 +62,7 @@ build = [
     "wheel",
 ]
 dev = [
-    "autopep8",
+    "autopep8~=2.0",
     "pre-commit",
     "pytest",
     "ruff==0.2.1",
@@ -82,27 +82,6 @@ yt-dlp = "yt_dlp:main"
 [project.entry-points.pyinstaller40]
 hook-dirs = "yt_dlp.__pyinstaller:get_hook_dirs"
 
-[tool.hatch.envs.default]
-dependencies = ["yt-dlp[dev]"]
-
-[tool.hatch.envs.default.scripts]
-install = "pre-commit install"
-lint = "ruff check {args:.}"
-format = "autopep8 {args:--in-place .}"
-tests = "python -m devscripts.run_tests {args}"
-
-[[tool.hatch.envs.default.overrides.env.TEST_ALL.matrix.value]]
-python = [
-    "3.8",
-    "3.9",
-    "3.10",
-    "3.11",
-    "3.12",
-    "pypy3.8",
-    "pypy3.9",
-    "pypy3.10",
-]
-
 [tool.hatch.build.targets.sdist]
 include = [
     "/yt_dlp",
@@ -113,7 +92,6 @@ include = [
     "/LICENSE",  # included as license
     "/pyproject.toml",  # included by default
     "/README.md",  # included as readme
-    "/setup.cfg",
     "/supportedsites.md",
 ]
 exclude = ["/yt_dlp/__pyinstaller"]
@@ -141,6 +119,31 @@ artifacts = ["/yt_dlp/extractor/lazy_extractors.py"]
 path = "yt_dlp/version.py"
 pattern = "_pkg_version = '(?P<version>[^']+)'"
 
+[tool.hatch.envs.default]
+features = ["dev"]
+
+[tool.hatch.envs.default.scripts]
+install = "pre-commit install"
+lint = "ruff check --fix {args:.}"
+format = "autopep8 --in-place {args:.}"
+check = [
+    "ruff check {args:.}",
+    "autopep8 --diff {args:.}",
+]
+test = "python -m devscripts.run_tests {args}"
+
+[[tool.hatch.envs.default.overrides.env.TEST_ALL.matrix.value]]
+python = [
+    "3.8",
+    "3.9",
+    "3.10",
+    "3.11",
+    "3.12",
+    "pypy3.8",
+    "pypy3.9",
+    "pypy3.10",
+]
+
 [tool.ruff]
 required-version = "0.2.1"
 line-length = 120
@@ -156,7 +159,7 @@ select = [
     "E", # pycodestyle errors
     "W", # pycodestyle warnings
     "F", # pyflakes
-    "I", # import order
+    # "I", # import order
 ]
 
 [tool.ruff.lint.per-file-ignores]
@@ -173,7 +176,48 @@ relative-imports-order = "closest-to-furthest"
 [tool.autopep8]
 max_line_length = 120
 recursive = true
+exit-code = true
 jobs = 0
+select = [
+    "E11",
+    "E121",
+    "E122",
+    "E123",
+    "E124",
+    "E125",
+    "E126",
+    "E127",
+    "E128",
+    "E129",
+    "E131",
+    "E133",
+    "E20",
+    "E211",
+    "E22",
+    "E224",
+    "E225",
+    "E226",
+    "E227",
+    "E228",
+    "E231",
+    "E241",
+    "E242",
+    "E251",
+    "E252",
+    "E26",
+    "E265",
+    "E266",
+    "E27",
+    "E301",
+    "E302",
+    "E303",
+    "E304",
+    "E305",
+    "E306",
+    "E502",
+    "W391",
+    "W504",
+]
 
 [tool.pytest.ini_options]
 addopts = "-ra -v --strict-markers"

yt_dlp/extractor/lefigaro.py

@@ -13,7 +13,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
 
     _TESTS = [{
         'url': 'https://video.lefigaro.fr/embed/figaro/video/les-francais-ne-veulent-ils-plus-travailler-suivez-en-direct-le-club-le-figaro-idees/',
-        'md5': 'e94de44cd80818084352fcf8de1ce82c',
+        'md5': 'a0c3069b7e4c4526abf0053a7713f56f',
         'info_dict': {
             'id': 'g9j7Eovo',
             'title': 'Les Français ne veulent-ils plus travailler ? Retrouvez Le Club Le Figaro Idées',
@@ -26,7 +26,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
         },
     }, {
         'url': 'https://video.lefigaro.fr/embed/figaro/video/intelligence-artificielle-faut-il-sen-mefier/',
-        'md5': '0b3f10332b812034b3a3eda1ef877c5f',
+        'md5': '319c662943dd777bab835cae1e2d73a5',
         'info_dict': {
             'id': 'LeAgybyc',
             'title': 'Intelligence artificielle : faut-il s’en méfier ?',
@@ -41,7 +41,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
 
     _WEBPAGE_TESTS = [{
         'url': 'https://video.lefigaro.fr/figaro/video/suivez-en-direct-le-club-le-figaro-international-avec-philippe-gelie-9/',
-        'md5': '3972ddf2d5f8b98699f191687258e2f9',
+        'md5': '6289f9489efb969e38245f31721596fe',
         'info_dict': {
             'id': 'QChnbPYA',
             'title': 'Où en est le couple franco-allemand ? Retrouvez Le Club Le Figaro International',
@@ -55,7 +55,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
         },
     }, {
         'url': 'https://video.lefigaro.fr/figaro/video/la-philosophe-nathalie-sarthou-lajus-est-linvitee-du-figaro-live/',
-        'md5': '3ac0a0769546ee6be41ab52caea5d9a9',
+        'md5': 'f6df814cae53e85937621599d2967520',
         'info_dict': {
             'id': 'QJzqoNbf',
             'title': 'La philosophe Nathalie Sarthou-Lajus est l’invitée du Figaro Live',
@@ -73,7 +73,8 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
 
-        player_data = self._search_nextjs_data(webpage, display_id)['props']['pageProps']['pageData']['playerData']
+        player_data = self._search_nextjs_data(
+            webpage, display_id)['props']['pageProps']['initialProps']['pageData']['playerData']
 
         return self.url_result(
             f'jwplatform:{player_data["videoId"]}', title=player_data.get('title'),

yt_dlp/extractor/magellantv.py

@@ -28,12 +28,24 @@ class MagellanTVIE(InfoExtractor):
             'tags': ['Investigation', 'True Crime', 'Justice', 'Europe'],
         },
         'params': {'skip_download': 'm3u8'},
+    }, {
+        'url': 'https://www.magellantv.com/watch/celebration-nation',
+        'info_dict': {
+            'id': 'celebration-nation',
+            'ext': 'mp4',
+            'tags': ['Art & Culture', 'Human Interest', 'Anthropology', 'China', 'History'],
+            'duration': 2640.0,
+            'title': 'Ancestors',
+        },
+        'params': {'skip_download': 'm3u8'},
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
         webpage = self._download_webpage(url, video_id)
-        data = self._search_nextjs_data(webpage, video_id)['props']['pageProps']['reactContext']['video']['detail']
+        data = traverse_obj(self._search_nextjs_data(webpage, video_id), (
+            'props', 'pageProps', 'reactContext',
+            (('video', 'detail'), ('series', 'currentEpisode')), {dict}), get_all=False)
         formats, subtitles = self._extract_m3u8_formats_and_subtitles(data['jwpVideoUrl'], video_id)
 
         return {