📜 Contributions by @karlicoss at [[python]]
Agora nodes contain community contributions with titles or topics that match your search. x
📜 programming/python/python.md (contribution by @karlicoss) ☆ 📎

Table of Contents

* testing [[testing]]

[2020-03-27] Hypothesis has a set of Pandas strategies for generating data [[pandas]]

https://www.hillelwayne.com/talks/beyond-unit-tests/

[2020-05-05] python - Run code before and after each test in py.test? - Stack Overflow

py.test fixtures are a technically adequate method to achieve your purpose.

You just need to define a fixture like that:

@pytest.fixture(autouse=True)
def run_around_tests():

[2019-08-02] python - How should I structure a pytest test only package? - Stack Overflow [[testing]] [[project]] [[pip]]

https://stackoverflow.com/questions/48111426/how-should-i-structure-a-pytest-test-only-package/48350323#48350323
distribute tests with package..

doctest?

pytest can detect recursion [[pytest]]

* profiling [[performance]]

[2020-03-08] what-studio/profiling: An interactive continuous Python profiler

https://github.com/what-studio/profiling 

profiling live-profile webserver.py

ok, this is really nice and easy to use

[2020-07-28] kinda unmaintainerd though?

[2019-07-20] profiling with cprof

python3 -m cProfile -o stats.prof script.py
snakeviz stats.prof

[2020-01-13] rkern/lineprofiler: Line-by-line profiling for Python

https://github.com/rkern/line_profiler

[2018-03-10] profiling

python3 -m cProfile -s tottime process.py | less
sudo apt install kcachegrind
pip3 install --user pyprof2calltree

https://julien.danjou.info/guide-to-python-profiling-cprofile-concrete-case-carbonara/
python -m cProfile -o myscript.cprof myscript.py
pyprof2calltree -k -i myscript.cprof

ipython?

timeit

* packaging [[pip]]

[2020-04-02] minimal-setup-py/setup.py at master · maet3608/minimal-setup-py

https://github.com/maet3608/minimal-setup-py/blob/master/setup.py

install only dependencies: pip3 install --user -e .

install multiple packages at once

ok, apparently pip does use some sort of ‘set compiling’, so worth doing that
https://github.com/pypa/pip/issues/988

[2020-04-24] local PIP dependency: if the repo is in /path/to/repo, put HPI@git+file://DUMMY/path/to/repo@master in setup.py [[pip]]

no idea why you need DUMMY bit, must be some parsing quirk :shrug:
also, sometimes it needs //DUMMY, sometimes ///DUMMY ???

[2019-12-15] bast/pypi-howto: How to publish Python packages on PyPI. [[pypi]]

https://github.com/bast/pypi-howto

[2019-07-02] Testing & Packaging · Homepage of Hynek Schlawack

https://hynek.me/articles/testing-packaging/ 

Your tests do not run against the package as it will be installed by its users. They run against whatever the situation in your project directory is.

[2019-07-25] o

To achieve that, you just move your packages into a src directory and add a where argument to find_packages() in your setup.py:

setup(
    [...]
    packages=find_packages(where="src"),
    package_dir={"": "src"},
)

releasing on pypi [[pypi]]

ok, that seems to be way easier 

hatch build --clean --verbose
hatch release --strict "$@"

* debugging [[debug]]

[2019-04-09] with ipdb.launch_ipdb_on_exception() [[habit]]

[2019-12-05] python3.7 has a breakpoint() builtin + more https://hackernoon.com/python-3-7s-new-builtin-breakpoint-a-quick-tour-4f1aebc444c

ipdb pp for pretty print [[habit]]

* bad things about python

  • lambdas suck
  • scoping (i.e. only local and global scopes)
  • GIL makes the concurrency very annoying at times
  • no static typing (although it’s kinda okay now with annotations and #mypy)

* static analysers

monkeytype – automatic typing annotations generation [[mypy]]

would be nice to have static analysis that checks you are not using global variables, etc.

however there are not any tools which do that…

bandit – security checks [[security]]

https://github.com/openstack/bandit

[2021-01-16] wouldn’t say it’s super useful… might detect occasional eval or whatever, but won’t help with more sophisticated stuff

* libraries

[2019-04-02] peopledoc/workalendar: Worldwide holidays and workdays computational toolkit.

https://github.com/peopledoc/workalendar

[2018-12-09] Altair: Declarative Visualization in Python — Altair 2.3.0 documentation [[viz]]

https://altair-viz.github.io/index.html

* Import system

figure it out – very confusing!

[2017-12-17] from . import module

local import

[2020-01-17] Alone Djangonaut – How python’s import machinery works

https://manikos.github.io/how-pythons-import-machinery-works 

origin

[2019-06-29] python - Relative imports for the billionth time - Stack Overflow

https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time 

.. are only relative in a package
However, if your module's name is __main__, it is not considered to be in a package. Its name has no dots, and therefore you cannot use from .. import statements inside it. If you try to do so, you will get the "relative-import in non-package" error.

[2019-06-11] relative imports are discouraged

* datetime handling [[datetime]]

[2018-09-04] utcfromtimestamp – returns timestamp, unaware; fromtimestamp – returns offset + timestamp, unaware

In : datetime.fromtimestamp(123)
Out: datetime.datetime(1970, 1, 1, 4, 2, 3)

In : datetime.utcfromtimestamp(123)
Out: datetime.datetime(1970, 1, 1, 0, 2, 3)

[2018-09-04] ok, seems that it’s better to use pytz.utc.localize(tz_unaware_datetime)

[2020-05-02] zachwill/moment: Dealing with dates in Python shouldn’t have to suck.

# Create a moment with words in it
moment.date("December 18, 2012")

[2020-05-02] zachwill/moment: Dealing with dates in Python shouldn’t have to suck.

# Create a moment that would normally be pretty hard to do
moment.date("2 weeks ago")

* logging [[logging]]

[2018-07-05] creating top level logger is always bad (may happen before configuring). use factory method [[logging]]

* docs

pydoc

-k: keyword search
-g: gui

ipython: ? and ??

* ipdb [[ipdb]]

related #repl

[2019-04-09] python - Use IPython magic functions in ipdb shell - Stack Overflow

https://stackoverflow.com/questions/16184487/use-ipython-magic-functions-in-ipdb-shell 

shell.find_line_magic('cpaste')()

—————————————————

[2019-09-22] Python3 f-strings - A complete guide to the most well received feature of Python 3. /r/Python

That’s a part of python’s data model like `len`, `reversed`, `iter`, …:
`datetime` simply defines [`datetime.__format__`](https://docs.python.org/3/library/datetime.html#datetime.date.__format__). The method [is explained here](https://docs.python.org/3/reference/datamodel.html#object.__format__).

[2019-05-11] To yield or not to yield [[yield]]

https://barahilia.github.io/blog/computers/2017/01/04/to-yield-or-not-to-yield.html 

Finally we came to this. To me it is more important that generators make code more elegant. Performance is usually relevant to smaller hot spots.
The idea is simple: in comparison to list interface where computation results are first saved and are returned only at the end, “yielding” could reach faster code and reduce memory usage. As a quick simulation we may use %timeit magic command in IPython with the following two functions:

[2019-07-30] good point about separating validation and generation

AST ast.dump(ast.parse("'a' in 'aa' in 'aaa'"))

use for – else [[habit]]

[2018-06-18] log optimizers i used

[2018-06-11] optimizing python CLI tools https://files.bemusement.org/talks/OSDC2008-FastPython/

interpreter – about 10ms
lazy imports
profile slow imports

parser.addargument(‘rest, nargs=argparse.REMAINDER) [[habit]]

process remaining arguments

awesome-python https://github.com/vinta/awesome-python

attrs – could be interesting… might solve stupid namedtuple addition etc.
https://github.com/python-attrs/attrs

This gives you the power to use actual classes with actual types in your code instead of confusing tuples or confusingly behaving namedtuples. Which in turn encourages you to write small classes that do one thing well. Never again violate the single responsibility principle just because implementing init et al is a painful drag.

should definitely subscribe to its feed and go through again

https://github.com/vinta/awesome-python#command-line-tools

pretty cool https://github.com/timofurrer/try

cool https://github.com/dbcli/mycli

hmm try that? http://docopt.org/

hmm that’s interesting, visual scraping https://github.com/scrapinghub/portia

https://www.reddit.com/r/coolgithubprojects/

?? https://python.libhunt.com/

do not use replace tzinfo, always localize.. [[habit]]

[2019-01-01] multiline string – lstrip(‘\n’) for nicer formatting!

[2019-03-20] Tweet from Vladislav Isenbaev (@isenbaev), at Mar 20, 03:48: pickle просто скотски медленный, если что - кратно медленнее даже json’а

<https://twitter.com/isenbaev/status/1108213595920166912 >

[2019-08-11] multiprocessing vs multithreading vs asyncio in Python 3.4 - Stack Overflow [[concurrency]] [[performance]]

https://stackoverflow.com/questions/27435284/multiprocessing-vs-multithreading-vs-asyncio-in-python-3-4 

if io_bound:
    if io_very_slow:
        print("Use Asyncio")
    else:
        print("Use Threads")
else:
    print("Multi Processing")

[2020-01-03] adamchainz/patchy: Patch the inner source of python functions at runtime. [[lisp]]

https://github.com/adamchainz/patchy#patchy 

Patch the inner source of python functions at runtime.

[2019-12-31] PyPy.js: Python in the web browser | Hacker News [[js]] [[webext]]

https://news.ycombinator.com/item?id=17819138

[2018-06-22] difflib.unified_diff is suuuuper slow…

[2018-04-21] scrapy debugging [[scrapy]]

scrapy shell

withsecrets scrapy shell

from scrapy.http import FormRequest
request=FormRequest(url=’http://www.quantified-mind.com/login’,formdata={‘Email’: USER,’Password’:PASSWORD,})
fetch(request) – redirects by default

sets response variable

you can do view(response) to view in browser
mind that it might be different from browsing in chrome. because of user agent?

[2018-08-30] python’s for..else makes in ‘syntactially’ look like a total function

NamedTuple

use typing.NamedTuple

watch out for default add and mul, it acts as adding pairs

you can’t easily overload init, so make types of fields robust. Use a helper method instead

def N(*args, **kwargs):
reurn NamedTuple(whatevs)

Use contextmanager

  • @contextmanager; prepare; yield exactly once, then shutdown
  • contextlib.redirect_stdout
    meh. I think it patches sys module? so it redirects sys.stdout calls only; but not all stdout

[2018-12-08] get current function/method name

def fname():
import inspect
return inspect.stack()[1][3]

traitlets?

[2019-04-30] What’s the difference between marshmallow and sql-alchemy? /r/flask

Marshmallow “converts” (deserializes) dicts to SQLAlchemy models or serializes SQLAlchemy models to dicts.
SQLAlchemy is an ORM. It maps database schema (tables) and data to Python objects.
The two packages complement each other. They cannot and do not replace each others’ functionality.
Flask-Marshmallow gives you a convenient interface to Marshmallow under Flask.

[2019-06-29] dominatepp/.travis.yml at master · karlicoss/dominatepp [[ci]]

https://github.com/karlicoss/dominatepp/blob/master/.travis.yml
hmm, minimalistic travis is not so bad..

[2019-08-12] use re.VERBOSE for inline comments in regexes

[2019-10-21] python - What do (lambda) function closures capture? - Stack Overflow [[plt]]

https://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture 

Scoping in Python is dynamic and lexical. A closure will always remember the name and scope of the variable, not the object it's pointing to.

[2019-12-30] argparse — Parser for command-line options, arguments and sub-commands — Python 3.8.1 documentation

https://docs.python.org/3/library/argparse.html > 

@dataclass
(init=False)
class FancyPath:
    path: Path
    mtime: float
    def __init__(self, path: Path):
        self.path = path
        self.mtime = path.stat().st_mtime

right, I suppose it’s type to use dataclasses

[2020-04-11] Traps for the Unwary in Python’s Import System — Nick Coghlan’s Python Notes 1.0 documentation [[python]]

http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-init-py-trap

[2020-05-13] amontalenti/elements-of-python-style: Goes beyond PEP8 to discuss what makes Python code feel great. A Strunk & White for Python. [[python]]

Use parens (...) for fluent APIs

[2020-05-18] The Definitive Guide to Python import Statements | Chris Yeh [[python]]

[2020-08-09] resize ipython notebook output window - Stack Overflow

You can toggle the scroll window in the main menu of the notebook
Cell -> Current Outputs -> Toggle Scrolling

[2020-01-26] Not really. Enable something like MyPy or PyType on a big enough codebase with z… | Hacker News

https://news.ycombinator.com/item?id=21902827 

Not really. Enable something like MyPy or PyType on a big enough codebase with zero explicit annotation and it'll already find plenty of bugs and unhandled cases from the inferred types alone. Some of these are stuff a strong IDE may catch too (using the wrong function name, passing wrong number of args, etc), but some other ones are actually deeper in the code.

So already, with zero annotations, you already get value out, let alone once you type a few tricky variables that are harder for Python to track.

[2020-07-21] tmbo/questionary: Python library to build pretty command line user prompts ✨Easy to use multi-select lists, confirmations, free text prompts … [[tui]]

[2020-09-07] Good Integration Practices — pytest documentation

tests in a separate dir vs tests in a subpackage

[2019-07-23] Python Scatterplot Matrix | Plotly [[viz]]

https://plot.ly/python/splom/

[2019-08-18] box/flaky: Plugin for nose or pytest that automatically reruns flaky tests. [[pytest]]

https://github.com/box/flaky

[2019-03-05] ipython -pylab [[ipython]]

[2018-12-23] lambdalisue/jupyter-vim-binding: Jupyter meets Vim. Vimmer will fall in love. [[vim]] [[jupyter]]

https://github.com/lambdalisue/jupyter-vim-binding

[2019-04-13] eh, I’m using emacs now…

[2019-05-10] Is SQLAlchemy recommended for a highly transactional, large volume database? /r/Python

Yes. I've written a few systems using SQLAlchemy that see 50-100k writes  and 200k or so reads per second, and as long as you're using the Core everything works fine. Also allows you to use the ORM for ease-of-development on bits that don't need performance.

SQLAlchemy is probably the one library in which I've never reached the "wait, I need to do something and can't" point. Every single weird edge case I've had to handle has been supported either natively or by being able to access the raw interfaces directly in a clean way. Fantastic library, can't recommend it more highly.

Also, the lead dev is extremely responsive on the google group and is more than happy to answer questions.

[2020-02-16] Command Line Scripts — Python Packaging Tutorial [[pip]]

https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point

🕒 Last updated 2021-05-16 01:16:22

Loading pushes...

✨ AI Synthesis Mistral Gemini ChatGPT Claude x

Expanding this section will automatically generate an AI synthesis of the contributions in this node.