domain_utils

https://img.shields.io/pypi/v/domain_utils.svg CI Documentation Status

A collection of util functions for extracting domains from urls.

Repo: https://github.com/openwpm/domain_utils

Install:

pip install domain-utils

Use:

import domain_utils as du
# Return just the url `my.domain.cloudfront.net/a/path/to/a/file.html`
du.stem_url('https://my.domain.cloudfront.net/a/path/to/a/file.html?a=1')
# Return just the eTLD+1 `domain.cloudfront.net`
du.get_etld1('https://my.domain.cloudfront.net/a/path/to/a/file.html?a=1')
# Get the port `5000`
du.get_port('https://localhost:5000/a/path/to/a/file.html?a=1')
# Get the scheme `wss`
du.get_scheme('wss://somedomain.example.com/a/path/to/a/ws')

This package was originally extracted from openwpm-utils.

Community Participation Guidelines

This project is governed by Mozilla’s code of conduct and etiquette guidelines.

For more details, please read the Mozilla Community Participation Guidelines.

For more information on how to report violations of the Community Participation Guidelines, please read our How to Report page.

History

0.8.0 (2026-09-05)

  • Relax the tldextract pin from ==2.2.2 to >=5.3 (#32). The pinned version cached the public suffix list inside its own installed package directory and refreshed it with an unguarded check-then-act, so any consumer running more than one process could race and lose data. Upstream moved the cache out of the package directory in 3.1.0.

  • Drop support for Python 3.6 to 3.10; the minimum is now 3.11.

  • Ship type annotations. The package now carries a py.typed marker, so type checkers see the annotations, and the whole codebase is checked with pyright in strict mode. Passing extractor=None explicitly is now a type error as well as the ValueError it always raised; leave the argument out to get the automatically created extractor.

  • Replace CircleCI with GitHub Actions (#31). CI now runs a single just ci inside the nix shell, so a red build reproduces locally with the same command and the same tool versions.

  • Replace the Makefile with a justfile. Every target was a .PHONY command alias, which is a command runner’s job rather than make’s.

  • The test suite now fails below 100% statement and branch coverage (#27).

  • Fix the docs build, and point urls at openwpm/domain_utils. Install instructions use the domain-utils project name (#30).

  • Publish to PyPI from a version tag using trusted publishing.

  • Fix url parsing on modern CPython. urlparse no longer refuses to read host:port as a scheme, which meant localhost:8000 and example.com:8080 lost their port, and 127.0.0.1:8080/a?b=1 lost its host entirely.

  • Fix scheme_default=None, which raised AttributeError on Python 3.11 to 3.13 because urlparse gained WHATWG scheme stripping.

  • stem_url now returns the url exactly as it was passed in when return_unparsed is set and the scheme is not parsed, rather than an internally rewritten form.

  • stem_url no longer appends a stray trailing slash to a scheme-less url that already has a path, e.g. 10.0.0.1:80/a/b.html. A scheme-less bare host, e.g. example.com, still gets one.

0.7.1 (2020-04-10)

Fix building on readthedocs.

0.7.0 (2020-04-10)

Thanks to new contributor @yabirgb for two PRs (#20 and #25) in this release.

API changes: #26 renamed get_stripped_url to stem_url, and get_ps_plus_1 to get_etld1. Old method names will continue to work though. #22 updated keyword arguments to get_stripped_url - default behavior is basically the same.

  • API changes (#26 and #22)

  • Support parsing ws/wss urls (#22)

  • Add get_port method (#25)

  • Add get_scheme method (#20)

  • Correct license declaration in setup.py (#24)

0.6.0 (2020-04-06)

  • Use tldextract for parsing domains (#12)

  • Use numpy style docstrings

  • Support case of no scheme and port in URL (#13)

0.5.0 (2020-04-03)

  • Remove support for python 3.5

  • Handle more cases in get_stripped_url and change default behavior:

    • handle a lack of scheme

    • boolean flag to return or not non http urls - default is to return them which is a change of behavior as previously they would not return

    • Use netloc by default instead of hostname with a boolean flag to use hostname.

0.4.0 (2020-03-25)

  • Remove py27 support

0.3.0 (2020-03-25)

  • Restore py27 support.

  • Last version with py27 support.

  • Remove tox

0.2.0 (2020-03-24)

Contents