.github/PULL_REQUEST_TEMPLATE.md

@@ -28,7 +28,7 @@ Fixes #
 ### Before submitting a *pull request* make sure you have:
 - [ ] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
 - [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
-- [ ] Checked the code using `hatch run check` (or ran `hatch run install` before committing) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)
+- [ ] Checked the code with [ruff](https://docs.astral.sh/ruff/) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)
 
 ### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
 - [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)

.github/workflows/quick-test.yml

@@ -20,8 +20,8 @@ jobs:
       run: |
         python3 -m yt_dlp -v || true
         python3 ./devscripts/run_tests.py core
-  check:
+  linter:
-    name: Code check
+    name: Linter
     if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ubuntu-latest
     steps:
@@ -32,6 +32,4 @@ jobs:
     - name: Make lazy extractors
       run: python3 ./devscripts/make_lazy_extractors.py
     - name: Run ruff
-      run: ruff check --output-format github .
+      run: ruff check . --output-format github
-    - name: Run autopep8
-      run: autopep8 --diff .

.gitignore

@@ -64,7 +64,7 @@ cookies
 # Python
 *.pyc
 *.pyo
-.*_cache
+.pytest_cache
 wine-py2exe/
 py2exe.log
 build/

.pre-commit-config.yaml

@@ -1,8 +1,13 @@
 repos:
 - repo: local
   hooks:
-  - id: check
+  - id: linter
-    name: code check
+    name: linter
-    entry: hatch run check
+    entry: hatch run lint
+    language: system
+    types: [python]
+  - id: format
+    name: format
+    entry: hatch run format --diff
     language: system
     types: [python]

CONTRIBUTING.md

@@ -134,31 +134,35 @@ We follow [youtube-dl's policy](https://github.com/ytdl-org/youtube-dl#can-you-a
 
 # DEVELOPER INSTRUCTIONS
 
-Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases), get them via [the other installation methods](README.md#installation) or directly run it using `python -m yt_dlp`.
+Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases) or get them via [the other installation methods](README.md#installation).
 
-`yt-dlp` uses [`hatch`](<https://hatch.pypa.io>) as a project management tool.
+To quickly run yt-dlp as a developer, you don't need to build anything either. Simply execute
-You can easily install it using [`pipx`](<https://pipx.pypa.io>) via `pipx install hatch`, or else via `pip` or your package manager of choice.
 
-If you plan on contributing to `yt-dlp`, you are required to run
+```shell
+$ python -m yt_dlp
+```
+
+To properly interact with `yt-dlp` as a developer, you should use [`hatch`](<https://hatch.pypa.io>) (easily installed using [`pipx`](<https://pipx.pypa.io>)).
+After installing, you can use `hatch shell` to create a shell with all dependencies as well as `yt-dlp` installed.
+
+If you plan on contributing to `yt-dlp`, run
 
 ```shell
 $ hatch run install
 ```
 
-before creating commits so that the pre-commit hooks are properly initialized.
+to prepare a full development environment and install `pre-commit`.
-After this you can use `hatch shell` to enable a virtual environment that has development dependencies as well as `yt-dlp` installed.
 
-Scripts can be used to run simple tasks, like linting or testing, without having to run `hatch shell` first:
+To run all the available core tests, use:
 
 ```shell
-$ hatch run lint
+$ hatch run tests core
-$ hatch run format
-$ hatch run test
 ```
 
-You can run scripts for all available and supported python versions sequentially by setting the `TEST_ALL` variable:
+You can also run tests for all installed python versions sequentially by setting the `TEST_ALL` variable, like so:
+
 ```shell
-$ TEST_ALL=1 hatch run test core
+$ TEST_ALL=1 hatch run tests core
 ```
 
 See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases.
@@ -239,16 +243,17 @@ After you have ensured this site is distributing its content legally, you can fo
             }
     ```
 1. Add an import in [`yt_dlp/extractor/_extractors.py`](yt_dlp/extractor/_extractors.py). Note that the class name must end with `IE`.
-1. Run `hatch run test YourExtractor`. This *may fail* at first, but you can continually re-run it until you're done. Upon failure, it will output the missing fields and/or correct values which you can copy. If you decide to add more than one test, the tests will then be named `YourExtractor`, `YourExtractor_1`, `YourExtractor_2`, etc. Note that tests with an `only_matching` key in the test's dict are not included in the count. You can also run all the tests in one go with `YourExtractor_all`
+1. Run `hatch run tests YourExtractor`. This *may fail* at first, but you can continually re-run it until you're done. Upon failure, it will output the missing fields and/or correct values which you can copy. If you decide to add more than one test, the tests will then be named `YourExtractor`, `YourExtractor_1`, `YourExtractor_2`, etc. Note that tests with an `only_matching` key in the test's dict are not included in the count. You can also run all the tests in one go with `YourExtractor_all`
 1. Make sure you have at least one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the particular test is disabled from running.
 1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L119-L440). Add tests and code for as many as you want.
 1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions), passes [ruff](https://docs.astral.sh/ruff/tutorial/#getting-started) code checks and is properly formatted:
 
     ```shell
-    $ hatch run check
+    $ hatch run lint
+    $ hatch run format
     ```
 
-    You can use `hatch run lint` and `hatch run format` to automatically fix problems.
+    You can use `hatch run lint --fix` to automatically fix problems.
 
 1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.8 and above. Backward compatibility is not required for even older versions of Python.
 1. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files, [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:

Makefile

@@ -10,7 +10,7 @@ tar: yt-dlp.tar.gz
 # intended use: when building a source distribution,
 # make pypi-files && python3 -m build -sn .
 pypi-files: AUTHORS Changelog.md LICENSE README.md README.txt supportedsites \
-	        completions yt-dlp.1 pyproject.toml devscripts/* test/*
+	        completions yt-dlp.1 pyproject.toml setup.cfg devscripts/* test/*
 
 .PHONY: all clean install test tar pypi-files completions ot offlinetest codetest supportedsites
 
@@ -24,7 +24,7 @@ clean-dist:
 	yt_dlp/extractor/lazy_extractors.py *.spec CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe yt_dlp.egg-info/ AUTHORS
 clean-cache:
 	find . \( \
-		-type d -name '.*_cache' -o -type d -name __pycache__ -o -name "*.pyc" -o -name "*.class" \
+		-type d -name .pytest_cache -o -type d -name __pycache__ -o -name "*.pyc" -o -name "*.class" \
 	\) -prune -exec rm -rf {} \;
 
 completion-bash: completions/bash/yt-dlp
@@ -38,13 +38,11 @@ MANDIR ?= $(PREFIX)/man
 SHAREDIR ?= $(PREFIX)/share
 PYTHON ?= /usr/bin/env python3
 
-# $(shell) and $(error) are no-ops in BSD Make and the != variable assignment operator is not supported by GNU Make <4.0
+# set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
-VERSION_CHECK != echo supported
+SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi)
-VERSION_CHECK ?= $(error GNU Make 4+ or BSD Make is required)
-CHECK_VERSION := $(VERSION_CHECK)
 
-# set markdown input format to "markdown-smart" for pandoc version 2+ and to "markdown" for pandoc prior to version 2
+# set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
-MARKDOWN != if [ "`pandoc -v | head -n1 | cut -d' ' -f2 | head -c1`" -ge "2" ]; then echo markdown-smart; else echo markdown; fi
+MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
 
 install: lazy-extractors yt-dlp yt-dlp.1 completions
 	mkdir -p $(DESTDIR)$(BINDIR)
@@ -67,7 +65,6 @@ uninstall:
 
 codetest:
 	ruff check .
-	autopep8 --diff .
 
 test:
 	$(PYTHON) -m pytest
@@ -76,17 +73,17 @@ test:
 offlinetest: codetest
 	$(PYTHON) -m pytest -k "not download"
 
-CODE_FOLDERS != find yt_dlp -type f -name '__init__.py' -exec dirname {} \+ | grep -v '/__' | sort
+CODE_FOLDERS := $(shell find yt_dlp -type d -not -name '__*' -exec sh -c 'test -e "$$1"/__init__.py' sh {} \; -print)
-CODE_FILES != for f in $(CODE_FOLDERS) ; do echo "$$f" | sed 's,$$,/*.py,' ; done
+CODE_FILES := $(shell for f in $(CODE_FOLDERS); do echo "$$f" | awk '{gsub(/\/[^\/]+/,"/*"); print $$1"/*.py"}'; done | sort -u)
 yt-dlp: $(CODE_FILES)
 	mkdir -p zip
 	for d in $(CODE_FOLDERS) ; do \
 	  mkdir -p zip/$$d ;\
 	  cp -pPR $$d/*.py zip/$$d/ ;\
 	done
-	(cd zip && touch -t 200001010101 $(CODE_FILES))
+	cd zip ; touch -t 200001010101 $(CODE_FILES)
 	mv zip/yt_dlp/__main__.py zip/
-	(cd zip && zip -q ../yt-dlp $(CODE_FILES) __main__.py)
+	cd zip ; zip -q ../yt-dlp $(CODE_FILES) __main__.py
 	rm -rf zip
 	echo '#!$(PYTHON)' > yt-dlp
 	cat yt-dlp.zip >> yt-dlp
@@ -130,7 +127,7 @@ completions/fish/yt-dlp.fish: $(CODE_FILES) devscripts/fish-completion.in
 	mkdir -p completions/fish
 	$(PYTHON) devscripts/fish-completion.py
 
-_EXTRACTOR_FILES != find yt_dlp/extractor -name '*.py' -and -not -name 'lazy_extractors.py'
+_EXTRACTOR_FILES = $(shell find yt_dlp/extractor -name '*.py' -and -not -name 'lazy_extractors.py')
 yt_dlp/extractor/lazy_extractors.py: devscripts/make_lazy_extractors.py devscripts/lazy_load_template.py $(_EXTRACTOR_FILES)
 	$(PYTHON) devscripts/make_lazy_extractors.py $@
 
@@ -142,14 +139,13 @@ yt-dlp.tar.gz: all
 		--exclude '*.pyo' \
 		--exclude '*~' \
 		--exclude '__pycache__' \
-		--exclude '.*_cache' \
+		--exclude '.pytest_cache' \
 		--exclude '.git' \
-		--exclude '__pyinstaller' \
 		-- \
 		README.md supportedsites.md Changelog.md LICENSE \
 		CONTRIBUTING.md Collaborators.md CONTRIBUTORS AUTHORS \
 		Makefile yt-dlp.1 README.txt completions .gitignore \
-		yt-dlp yt_dlp pyproject.toml devscripts test
+		setup.cfg yt-dlp yt_dlp pyproject.toml devscripts test
 
 AUTHORS:
 	git shortlog -s -n HEAD | cut -f2 | sort > AUTHORS

devscripts/run_tests.py

@@ -32,9 +32,9 @@ def run_tests(*tests, pattern=None, ci=False):
     if pattern:
         arguments.extend(['-k', pattern])
     if run_core:
-        arguments.extend(['-m', 'not download'])
+        arguments.extend(['--ignore', 'test/test_download.py'])
     elif run_download:
-        arguments.extend(['-m', 'download'])
+        arguments.extend(['test/test_download.py'])
     else:
         arguments.extend(
             f'test/test_download.py::TestDownload::test_{test}' for test in tests)

pyproject.toml

@@ -62,7 +62,7 @@ build = [
     "wheel",
 ]
 dev = [
-    "autopep8~=2.0",
+    "autopep8",
     "pre-commit",
     "pytest",
     "ruff==0.2.1",
@@ -82,6 +82,27 @@ yt-dlp = "yt_dlp:main"
 [project.entry-points.pyinstaller40]
 hook-dirs = "yt_dlp.__pyinstaller:get_hook_dirs"
 
+[tool.hatch.envs.default]
+dependencies = ["yt-dlp[dev]"]
+
+[tool.hatch.envs.default.scripts]
+install = "pre-commit install"
+lint = "ruff check {args:.}"
+format = "autopep8 {args:--in-place .}"
+tests = "python -m devscripts.run_tests {args}"
+
+[[tool.hatch.envs.default.overrides.env.TEST_ALL.matrix.value]]
+python = [
+    "3.8",
+    "3.9",
+    "3.10",
+    "3.11",
+    "3.12",
+    "pypy3.8",
+    "pypy3.9",
+    "pypy3.10",
+]
+
 [tool.hatch.build.targets.sdist]
 include = [
     "/yt_dlp",
@@ -92,6 +113,7 @@ include = [
     "/LICENSE",  # included as license
     "/pyproject.toml",  # included by default
     "/README.md",  # included as readme
+    "/setup.cfg",
     "/supportedsites.md",
 ]
 exclude = ["/yt_dlp/__pyinstaller"]
@@ -119,31 +141,6 @@ artifacts = ["/yt_dlp/extractor/lazy_extractors.py"]
 path = "yt_dlp/version.py"
 pattern = "_pkg_version = '(?P<version>[^']+)'"
 
-[tool.hatch.envs.default]
-features = ["dev"]
-
-[tool.hatch.envs.default.scripts]
-install = "pre-commit install"
-lint = "ruff check --fix {args:.}"
-format = "autopep8 --in-place {args:.}"
-check = [
-    "ruff check {args:.}",
-    "autopep8 --diff {args:.}",
-]
-test = "python -m devscripts.run_tests {args}"
-
-[[tool.hatch.envs.default.overrides.env.TEST_ALL.matrix.value]]
-python = [
-    "3.8",
-    "3.9",
-    "3.10",
-    "3.11",
-    "3.12",
-    "pypy3.8",
-    "pypy3.9",
-    "pypy3.10",
-]
-
 [tool.ruff]
 required-version = "0.2.1"
 line-length = 120
@@ -159,7 +156,7 @@ select = [
     "E", # pycodestyle errors
     "W", # pycodestyle warnings
     "F", # pyflakes
-    # "I", # import order
+    "I", # import order
 ]
 
 [tool.ruff.lint.per-file-ignores]
@@ -176,48 +173,7 @@ relative-imports-order = "closest-to-furthest"
 [tool.autopep8]
 max_line_length = 120
 recursive = true
-exit-code = true
 jobs = 0
-select = [
-    "E11",
-    "E121",
-    "E122",
-    "E123",
-    "E124",
-    "E125",
-    "E126",
-    "E127",
-    "E128",
-    "E129",
-    "E131",
-    "E133",
-    "E20",
-    "E211",
-    "E22",
-    "E224",
-    "E225",
-    "E226",
-    "E227",
-    "E228",
-    "E231",
-    "E241",
-    "E242",
-    "E251",
-    "E252",
-    "E26",
-    "E265",
-    "E266",
-    "E27",
-    "E301",
-    "E302",
-    "E303",
-    "E304",
-    "E305",
-    "E306",
-    "E502",
-    "W391",
-    "W504",
-]
 
 [tool.pytest.ini_options]
 addopts = "-ra -v --strict-markers"

yt_dlp/extractor/lefigaro.py

@@ -13,7 +13,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
 
     _TESTS = [{
         'url': 'https://video.lefigaro.fr/embed/figaro/video/les-francais-ne-veulent-ils-plus-travailler-suivez-en-direct-le-club-le-figaro-idees/',
-        'md5': 'a0c3069b7e4c4526abf0053a7713f56f',
+        'md5': 'e94de44cd80818084352fcf8de1ce82c',
         'info_dict': {
             'id': 'g9j7Eovo',
             'title': 'Les Français ne veulent-ils plus travailler ? Retrouvez Le Club Le Figaro Idées',
@@ -26,7 +26,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
         },
     }, {
         'url': 'https://video.lefigaro.fr/embed/figaro/video/intelligence-artificielle-faut-il-sen-mefier/',
-        'md5': '319c662943dd777bab835cae1e2d73a5',
+        'md5': '0b3f10332b812034b3a3eda1ef877c5f',
         'info_dict': {
             'id': 'LeAgybyc',
             'title': 'Intelligence artificielle : faut-il s’en méfier ?',
@@ -41,7 +41,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
 
     _WEBPAGE_TESTS = [{
         'url': 'https://video.lefigaro.fr/figaro/video/suivez-en-direct-le-club-le-figaro-international-avec-philippe-gelie-9/',
-        'md5': '6289f9489efb969e38245f31721596fe',
+        'md5': '3972ddf2d5f8b98699f191687258e2f9',
         'info_dict': {
             'id': 'QChnbPYA',
             'title': 'Où en est le couple franco-allemand ? Retrouvez Le Club Le Figaro International',
@@ -55,7 +55,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
         },
     }, {
         'url': 'https://video.lefigaro.fr/figaro/video/la-philosophe-nathalie-sarthou-lajus-est-linvitee-du-figaro-live/',
-        'md5': 'f6df814cae53e85937621599d2967520',
+        'md5': '3ac0a0769546ee6be41ab52caea5d9a9',
         'info_dict': {
             'id': 'QJzqoNbf',
             'title': 'La philosophe Nathalie Sarthou-Lajus est l’invitée du Figaro Live',
@@ -73,8 +73,7 @@ class LeFigaroVideoEmbedIE(InfoExtractor):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
 
-        player_data = self._search_nextjs_data(
+        player_data = self._search_nextjs_data(webpage, display_id)['props']['pageProps']['pageData']['playerData']
-            webpage, display_id)['props']['pageProps']['initialProps']['pageData']['playerData']
 
         return self.url_result(
             f'jwplatform:{player_data["videoId"]}', title=player_data.get('title'),

yt_dlp/extractor/magellantv.py

@@ -28,24 +28,12 @@ class MagellanTVIE(InfoExtractor):
             'tags': ['Investigation', 'True Crime', 'Justice', 'Europe'],
         },
         'params': {'skip_download': 'm3u8'},
-    }, {
-        'url': 'https://www.magellantv.com/watch/celebration-nation',
-        'info_dict': {
-            'id': 'celebration-nation',
-            'ext': 'mp4',
-            'tags': ['Art & Culture', 'Human Interest', 'Anthropology', 'China', 'History'],
-            'duration': 2640.0,
-            'title': 'Ancestors',
-        },
-        'params': {'skip_download': 'm3u8'},
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
         webpage = self._download_webpage(url, video_id)
-        data = traverse_obj(self._search_nextjs_data(webpage, video_id), (
+        data = self._search_nextjs_data(webpage, video_id)['props']['pageProps']['reactContext']['video']['detail']
-            'props', 'pageProps', 'reactContext',
-            (('video', 'detail'), ('series', 'currentEpisode')), {dict}), get_all=False)
         formats, subtitles = self._extract_m3u8_formats_and_subtitles(data['jwpVideoUrl'], video_id)
 
         return {