Pwntools doctest execution in Sphinx

[maxdiesel]

Doctest in Sphinx Write Up

Find test differential and test modules not included in execution of

$ PWNLIB_NOTERM=1 make clean doctest

Sphinx = 941 vs 967 => difference of 26 tests.

Comparison of the Sphinx results to Pwntools Library resulted in the following list of unreported tests.

About
Atexception
Commandline
Dynelf
Elf
Environment.pickle
Exception
Index
Install
Log
Replacements
Term
UI
Tubes/Serial
Util/Net
Util/Hashes
Install/Binutils
Install/Headers
Shellcraft/Common

Getting Sphinx to execute the previous list of tests

Current efforts of first finding the files lead me to

/binjitsu/pwnlib which contain .py and .pyc
/binjitsu/docs/source which contain .rst
/binjitsu/docs/build/doctree which contain .doctree

Attempt to include missing test include

sphinx-quickstart with --ext-doctest extension
sphinx-build
sys.path update to include '/binjitsu/pwnlib'

all of which have yet to yield the desired results.

[zachriggle]

Here's something that may help out a bit. The current documentation is generated via the automodule and module helpers for Sphinx.

By comparing the modules which are listed with these helpers, versus the files which have doctests in them, you should be able to identify some of the gaps. For example, pwnlib.tubes.buffer is not listed in the documentation, but does contain doctests.

$ grep -ri module:: docs | grep -o 'pwnlib.*' | sort | uniq
pwnlib
pwnlib.asm
pwnlib.atexception
pwnlib.atexit
pwnlib.constants
pwnlib.dynelf
pwnlib.elf
pwnlib.exception
pwnlib.gdb
pwnlib.log
pwnlib.memleak
pwnlib.replacements
pwnlib.rop
pwnlib.shellcraft
pwnlib.shellcraft.amd64
pwnlib.shellcraft.amd64.linux
pwnlib.shellcraft.arm
pwnlib.shellcraft.arm.linux
pwnlib.shellcraft.common
pwnlib.shellcraft.i386
pwnlib.shellcraft.i386.freebsd
pwnlib.shellcraft.i386.linux
pwnlib.shellcraft.thumb
pwnlib.shellcraft.thumb.linux
pwnlib.term
pwnlib.timeout
pwnlib.tubes
pwnlib.tubes.listen
pwnlib.tubes.process
pwnlib.tubes.remote
pwnlib.tubes.serialtube
pwnlib.tubes.sock
pwnlib.tubes.ssh
pwnlib.tubes.tube
pwnlib.ui
pwnlib.useragents
pwnlib.util.crc
pwnlib.util.cyclic
pwnlib.util.fiddling
pwnlib.util.hashes
pwnlib.util.iters
pwnlib.util.lists
pwnlib.util.misc
pwnlib.util.net
pwnlib.util.packing
pwnlib.util.proc
pwnlib.util.safeeval
pwnlib.util.web

Meanwhile, here's the list of everything that looks like it has a doctest in it:

$ grep -lri '>>>' pwnlib | sort | uniq
pwnlib/asm.py
pwnlib/commandline/shellcraft.py
pwnlib/constants/__init__.py
pwnlib/context/__init__.py
pwnlib/elf/__init__.py
pwnlib/gdb.py
pwnlib/memleak.py
pwnlib/rop.py
pwnlib/shellcraft/templates/amd64/linux/mov.asm
pwnlib/shellcraft/templates/amd64/linux/push.asm
pwnlib/shellcraft/templates/amd64/linux/syscall.asm
pwnlib/shellcraft/templates/amd64/mov.asm
pwnlib/shellcraft/templates/amd64/push.asm
pwnlib/shellcraft/templates/amd64/pushstr.asm
pwnlib/shellcraft/templates/arm/linux/syscall.asm
pwnlib/shellcraft/templates/arm/mov.asm
pwnlib/shellcraft/templates/arm/ret.asm
pwnlib/shellcraft/templates/__doc__
pwnlib/shellcraft/templates/i386/freebsd/mov.asm
pwnlib/shellcraft/templates/i386/freebsd/push.asm
pwnlib/shellcraft/templates/i386/linux/mov.asm
pwnlib/shellcraft/templates/i386/linux/push.asm
pwnlib/shellcraft/templates/i386/linux/syscall.asm
pwnlib/shellcraft/templates/i386/mov.asm
pwnlib/shellcraft/templates/i386/push.asm
pwnlib/shellcraft/templates/i386/pushstr.asm
pwnlib/shellcraft/templates/i386/stackhunter.asm
pwnlib/shellcraft/templates/thumb/linux/mov.asm
pwnlib/shellcraft/templates/thumb/linux/push.asm
pwnlib/shellcraft/templates/thumb/mov.asm
pwnlib/shellcraft/templates/thumb/push.asm
pwnlib/shellcraft/templates/thumb/pushstr.asm
pwnlib/timeout.py
pwnlib/tubes/buffer.py
pwnlib/tubes/process.py
pwnlib/tubes/remote.py
pwnlib/tubes/ssh.py
pwnlib/tubes/tube.py
pwnlib/useragents.py
pwnlib/util/crc/__init__.py
pwnlib/util/cyclic.py
pwnlib/util/fiddling.py
pwnlib/util/iters.py
pwnlib/util/lists.py
pwnlib/util/misc.py
pwnlib/util/packing.py
pwnlib/util/proc.py
pwnlib/util/safeeval.py
pwnlib/util/web.py

[zachriggle]

This pull request should also not be against this repository, but against https://github.com/binjitsu/binjitsu.

[maxdiesel]

Hi Zach,

I can talk tonight if you are able. I live in DC.

Darrell

Zach Riggle [email protected] wrote:

Here's something that may help out a bit. The current documentation is generated via the automodule and module helpers for Sphinx.

By comparing the modules which are listed with these helpers, versus the files which have doctests in them, you should be able to identify some of the gaps. For example, pwnlib.tubes.buffer is not listed in the documentation, but does contain doctests.

$ grep -ri module:: docs | grep -o 'pwnlib.*' | sort | uniq
pwnlib
pwnlib.asm
pwnlib.atexception
pwnlib.atexit
pwnlib.constants
pwnlib.dynelf
pwnlib.elf
pwnlib.exception
pwnlib.gdb
pwnlib.log
pwnlib.memleak
pwnlib.replacements
pwnlib.rop
pwnlib.shellcraft
pwnlib.shellcraft.amd64
pwnlib.shellcraft.amd64.linux
pwnlib.shellcraft.arm
pwnlib.shellcraft.arm.linux
pwnlib.shellcraft.common
pwnlib.shellcraft.i386
pwnlib.shellcraft.i386.freebsd
pwnlib.shellcraft.i386.linux
pwnlib.shellcraft.thumb
pwnlib.shellcraft.thumb.linux
pwnlib.term
pwnlib.timeout
pwnlib.tubes
pwnlib.tubes.listen
pwnlib.tubes.process
pwnlib.tubes.remote
pwnlib.tubes.serialtube
pwnlib.tubes.sock
pwnlib.tubes.ssh
pwnlib.tubes.tube
pwnlib.ui
pwnlib.useragents
pwnlib.util.crc
pwnlib.util.cyclic
pwnlib.util.fiddling
pwnlib.util.hashes
pwnlib.util.iters
pwnlib.util.lists
pwnlib.util.misc
pwnlib.util.net
pwnlib.util.packing
pwnlib.util.proc
pwnlib.util.safeeval
pwnlib.util.web

Meanwhile, here's the list of everything that looks like it has a doctest in it:

$ grep -lri '>>>' pwnlib | sort | uniq
pwnlib/asm.py
pwnlib/commandline/shellcraft.py
pwnlib/constants/__init__.py
pwnlib/context/__init__.py
pwnlib/elf/__init__.py
pwnlib/gdb.py
pwnlib/memleak.py
pwnlib/rop.py
pwnlib/shellcraft/templates/amd64/linux/mov.asm
pwnlib/shellcraft/templates/amd64/linux/push.asm
pwnlib/shellcraft/templates/amd64/linux/syscall.asm
pwnlib/shellcraft/templates/amd64/mov.asm
pwnlib/shellcraft/templates/amd64/push.asm
pwnlib/shellcraft/templates/amd64/pushstr.asm
pwnlib/shellcraft/templates/arm/linux/syscall.asm
pwnlib/shellcraft/templates/arm/mov.asm
pwnlib/shellcraft/templates/arm/ret.asm
pwnlib/shellcraft/templates/__doc__
pwnlib/shellcraft/templates/i386/freebsd/mov.asm
pwnlib/shellcraft/templates/i386/freebsd/push.asm
pwnlib/shellcraft/templates/i386/linux/mov.asm
pwnlib/shellcraft/templates/i386/linux/push.asm
pwnlib/shellcraft/templates/i386/linux/syscall.asm
pwnlib/shellcraft/templates/i386/mov.asm
pwnlib/shellcraft/templates/i386/push.asm
pwnlib/shellcraft/templates/i386/pushstr.asm
pwnlib/shellcraft/templates/i386/stackhunter.asm
pwnlib/shellcraft/templates/thumb/linux/mov.asm
pwnlib/shellcraft/templates/thumb/linux/push.asm
pwnlib/shellcraft/templates/thumb/mov.asm
pwnlib/shellcraft/templates/thumb/push.asm
pwnlib/shellcraft/templates/thumb/pushstr.asm
pwnlib/timeout.py
pwnlib/tubes/buffer.py
pwnlib/tubes/process.py
pwnlib/tubes/remote.py
pwnlib/tubes/ssh.py
pwnlib/tubes/tube.py
pwnlib/useragents.py
pwnlib/util/crc/__init__.py
pwnlib/util/cyclic.py
pwnlib/util/fiddling.py
pwnlib/util/iters.py
pwnlib/util/lists.py
pwnlib/util/misc.py
pwnlib/util/packing.py
pwnlib/util/proc.py
pwnlib/util/safeeval.py
pwnlib/util/web.py

---

Reply to this email directly or view it on GitHub:
#30 (comment)

[zachriggle]

Darrell

Check your inbox for mail from Melange

Zach

[maxdiesel]

Hey Zach,

I'm available to meet know if you are able. Call, text, or video chat

Darrell

Cell 202-725-5946
Gmail [email protected]

On Sun, Apr 5, 2015 at 6:07 PM, Zach Riggle [email protected]
wrote:

Darrell

Check your inbox for mail from Melange

Zach

—
Reply to this email directly or view it on GitHub
#30 (comment)
.