96 Commits

Author SHA1 Message Date
JP Smith
60fa4c2231 Fix memdump tests broken by dev-events (#377)
* fix memdump tests broken by dev-events

* switch to superset check to account for timeout differences
2017-07-12 18:11:08 -04:00
Yan
45d5b9f6d2 New state serialization (#376)
* Fix naming

* Separate storage from workspace

* Begin removing output generation from manticore.py

* Split up workspace and output

* Create a separation between output and workspace

* Get it to a working (hacky) state

* Start bringing state serialization into workspace

* More mcore->workspace moves

* Remove unused imports

* Update serializers; add interface

* move state saving to workspace

* can now save/load states and testcases

* Add redis as backend

* Implement streams in terms of values and vice versa

* Implement `ls` and move workspace loading to Workspace

* Better workspace initialization

* Fix how --workspace is parsed for tests

* update tests to new workspace indexes

* Fix state storage and handling

* Doc updates and extra error checking

* pep8 style changes

* Add Executor's 'sync'

* Remove previous workspace artifacts from Executor

* Comment out constraints check assert

* Centralize locking to new testcase ids

* Fix testcase generation

* pep8 updates

* propagate descriptor instead of path

* Add an in-memory state store

* Make memory maps be sortable

* Allow to produce an empty state

* Add workspace tests

* Make Memory.__cmp__() more robust
2017-07-12 17:50:24 -04:00
JP Smith
b6a71375cf Remove md5 tests and replace with set comparison (#374)
* remove md5 tests and replace with comparison

* fix directory path issue

* remove exact check as test is not deterministic

* fix argument assertion test and remove cdet_visited.txt

* allow extra addresses in actual to account for inter-machine differences
2017-07-07 15:43:52 -04:00
Mark Mossberg
e4102e5000 Catch and refactor MemoryExceptions (#370)
* initial

* Clean, only store address in message if not symbolic

If it's symbolic,it renders like

    Generating testcase No. 2 - Invalid symbolic memory access (mode:r) <<manticore.core.smtlib.expression.BitVecAnd object at 0x7fd6dcd14410>>

which is not useful output to the user.

* Format as hex

* Fix tests

* Rm comments
2017-06-30 17:47:55 -04:00
Garret Reece
7d3d9a9ed1 lsl.w in thumb mode fixed, plus tests (#350)
- blx <Rm> swaps mode appropriately
- asr.w and lsr.w in thumb mode handle immediates for op2 correctly
- unit tests for thumb mode behavior for shifting instructions
- unit tests check for correct mode on BX and BLX instructions
2017-06-27 13:52:20 -05:00
Garret Reece
e931992abb Add ldrd and strd instructions and tests (#348)
* LDRD and STRD implementation and tests
2017-06-26 17:22:59 -05:00
feliam
520a9be47d Dev - events (#341)
* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* forwarding events wip

* state setstate fix and setup_stack merge fix

* will_terminate_state fix and tests skipped

* Update all ConcretizeRegister and ConcretizeMemory

* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* forwarding events wip

* state setstate fix and setup_stack merge fix

* will_terminate_state fix and tests skipped

* Update all ConcretizeRegister and ConcretizeMemory

* Exceptions are crazy crazy crazy

* fix last merge

* Merge merge until it pass

* Instructions count default to 0

* will/did execute/emulate

* Delayed keybpoard interrupt now shutdowns nicely

* fix auto test generator x86

* Undo bad merge

* utterly hopeless

* basic working

* Fix merge bugs and github comments

* Remove unnecesary comment - github comments

* trace_item not used there

* model-platform and system.py fixed

* backup/restore to store/load -- cpu.instruction property

* Slightly better did/will naming and dynamic signal forwarding

* platform.constraints and cpu.instruction as properties

* Fix forward signals getattr

* set las decoded pc at decode_instruction() / reenable instruction_cache

* Signals name convention: did/will/on

* Forward normal signals

* Maintain last decoded pc in abstractcpu

* Changed context manager so it just wont raise interrupt

* Decree now forwards signals and sets constraints

* linux.SymbolicFile does not need to maintain constraints

* remove debbug print

* Assimilating some PR commets

* size_total == size

* better merge of manticore.py

* typo

* Forwarding only specified objects in signal arguments

* Fix few broken tests

* revert + merge

* remove some unused stuff from manticore()

* manticore context <-> executor context

* manticore context <-> executor context2

* context context context

* forgotten return

* Fix basix.arm

* arm bitwise fix

* fix context

* Comment 1

* Comment 2

* Comment 3

* Comment 4

* Comment 5

* Comment 6

* Fix (still needs refactor but it works) profiling

* Fix (still needs refactor but it works) profiling

* The forgotten bit

* Update tests to reflect current output

* Verbosity fix

* Fix verbosity test
2017-06-26 18:06:18 -03:00
Theofilos Petsios
e2c0414dca added _multiprocess_can_split_ directive (#351)
* added _multiprocess_can_split_ directive

* renamed Readme
2017-06-23 17:53:19 -04:00
Mark Mossberg
3ec9043b2e Fix verbosity=1 (#345)
* Only enable debug executor after verbosity 1

* Add regression test for checking uniform cli output for nonverbose mode
2017-06-21 17:48:37 -04:00
Garret Reece
7f550becc6 Add mrc and movt
* Add MOVT instruction

* Add new ARM operand types for MRC

* mrc implementation seems to work

* UXTB inst implemented, provisional implementations of LDREX and STREX

* Added unit tests for MRC, MOVT, and UXTB

* Added mode swapping from blx instruction on arm cpu

* Make tests pass
2017-06-13 13:21:19 -05:00
JP Smith
5ea4a87ddb Fix SymbolicMemoryException (#303)
* fix SymbolicMemoryException

* make simple_parse's visited.txt more correct
2017-06-12 15:28:57 -04:00
JP Smith
31b39ccf9d unskip win32 API test (#294) 2017-06-09 11:39:28 -04:00
JP Smith
c7f5600cd1 unskip basic stdin test (#293) 2017-06-09 10:47:55 -04:00
Mark Mossberg
5b5392e54d Implement user variadic model interface (#276)
* Add variadic decorator

* Add check for _variadic func attribute

* Expose @variadic at top level

* Use variable for the actual name of attr

* Update naming

* Use helper

* Maybe more pythonic?

* Add variadic to public API

* Add variadic decorator tests

* Clean

* Remove variadic parameter

* Add docstrings

* Document parameter

* Clean

* Better use of autodoc

* Update docstring

* Add todo comment
2017-05-31 15:48:19 -04:00
Mark Mossberg
8354ded7e0 Test hook api (#274) 2017-05-25 16:22:08 -04:00
Mark Mossberg
c2b0617f32 Check returncode of manticore (#273) 2017-05-25 16:04:34 -04:00
Yan
3c977719f0 Linux platform refactoring (#264)
* Clean up model syscall invocation
* Move read_string to Cpu
* move push/pop helpers to Cpu
* Reorg Linux initialization
* Update linux test
* fstat64 test harness
* assert read_string only considers concrete bytes
* Ensure that correct aliases exist during ctor
* Improve alias check
2017-05-23 15:30:58 -04:00
feliam
475f786f21 Fix x86.PUNPCKL emulation (#259)
* Fix x86.PUNPCKL emulation

* trace check wip

* PUNPCKH/L and Readlink /proc

* PUNPCK fixes

* remove debug traceback

* rename arguments insteag of commenting them

* fix comment
2017-05-23 15:21:48 -03:00
Mark Mossberg
8840e31fc2 Add strlen model (#260)
* add models

* wip strcmp tests

* t

* Add some tests

* Better asserts

* More pythonic

* Add effectivene null test

* init strlen

* pythonic range

* clean

* docstring

* tmp

* stuff

* stuff

* Complete test

* Add another test

* Doc

* clean

* Add symbolic arg check

* Fix tests

* Make arch generic

* Clean comment

* Add basic unconstrained test
2017-05-18 16:59:44 -04:00
Mark Mossberg
92eaf76236 Add strcmp model (#251)
* add models

* wip strcmp tests

* t

* Add some tests

* Better asserts

* More pythonic

* Add effectivene null test

* Handle symbolic pointer arguments
2017-05-17 11:44:10 -04:00
Mark Mossberg
7700112d9e Rm unused Slinux ctor params (#250)
* Rm unused Slinux ctor params

Fixes #109

* Add back constraitns

* Make argv and envp default params

* Use ternary

* nit

* Clean test_unicorn

* Correct envp and docstring
2017-05-12 11:35:59 -04:00
Mark Mossberg
e4a4916597 Rename os model terminology from "models" to "platforms" (#243)
* Rename

* rename in manticore.py

* rename in executor.py

* big rename

* big rename

* update changelog
2017-05-09 19:25:32 -04:00
Yan
3c9653d1d7 Update syscall/funcall implementation (#213)
* Wrap ABI semantics in its own class hierarchy

* Define a model invocation for syscalls and function calls

* Add unit tests for ABI

* Add a common base class for Platform models
2017-05-09 16:13:08 -04:00
Yan
395a40a646 Add memory tracing (#203)
* implement memory write tracing

* Comment updates

* Add memory trace tests

* make sure we ignore erroring writes

* Address comments

* remove superfluous return annotation
2017-05-08 14:32:54 -04:00
Mark Mossberg
44d365ff4c Formalize API for constraining a State (#232)
* Rename state.add to state.constrain

* Update all uses of state.constrain

* Rm check param

* Added changelog

* Update changelog

* Update

* Minor clean
2017-05-05 13:31:43 -04:00
Yan
50fd50ee12 Simplify linux model interfaces (#233) 2017-05-05 13:11:59 -04:00
Yan
64d1ed51f2 Decode instruction early (#205)
* Move PC verification to decode_instruction

* Decode instruction in executor

* Fix unicorn tests

* remove decoded_pc

* use issymbolic()
2017-05-04 17:36:39 -04:00
Yan
8d6bcadb37 Fix RSB, VLDMIA; Add RSC (#197)
* Fix RSB; Add RSC
* Fix vldmia semantics
* To trigger build
2017-04-27 16:36:23 -04:00
Mark Mossberg
fd3873f0a8 Fix ARM bitwise (LSL/LSR) (#174)
* Rm bogus asserts

* correct lsl_c

* Add test for lsl carry
2017-04-24 11:31:11 -04:00
Mark Mossberg
a52be7459d Arm add vldmia (#154)
* Add VLDMIA

* Add vldmia tests
2017-04-20 15:30:50 -04:00
Mark Mossberg
3e81cba8fb Fix linter issues on cpu (#144)
* quick linter stuff

* Fix cpu init in tests

* update cpu name
2017-04-19 10:29:05 -04:00
JP Smith
e32701f978 Refactor logging (#140)
* begin refactoring logging to use manticore object

* fix verbosity things

* fix logging when in scripts

* remove m.log_file now that it's no longer used

* remove unnecessary init_logging usage and unused log_debug variable

* accidentally deleted the wrong line

* re-hide init_logging

* remove old log_debug

* remove unnecessary API stuff and refactor around that

* re-introduce logging PR

* fix Popen

* lost a line in merge

* fix test_binaries

* Log to stdout instead of stderr

* implement mark's changes
2017-04-19 01:21:14 -05:00
feliam
a9711cf119 Graceful fallback if Z3 doesn't support optimization queries (#135)
* Configure Z3 depending on installed version

* Mocking for subprocess.check_output

* change get_value_fmt to private attribute _get_value_fmt

* Move memoized to utils

* Remove minimal version and simplify

* Re-add invalid versions/sanity checks

* Z3 version format checked on github

* New tests. Version is future proof

* Better logging
2017-04-18 18:40:06 -03:00
Mark Mossberg
05b7c1138b Fix memory._ceil, Fix linux.load (#134)
* Reimplement ceil so page aligned addresses are unaffected

* Lingering fix

* Fixup users of _ceil

* Correct memory tests
2017-04-13 18:45:18 -04:00
feliam
bbfb234cca Dev fix amd64 emulation bugs (#136)
* Failing tests

* Tha Fix

* Fix auto test

* PSLLDQ for 32b python?

* More descriptive temporary variable
2017-04-13 19:22:04 -03:00
Yan
0b850dba40 Make unicorn be pull-based (#97)
* Enable simple ARM register concretization for Unicorn

* Add canonical_registers property to abstractcpu

* cpu to self

* Check for regs_access better

* Emulate a single instruction

* Bypass capstone 3.0.4 arm bug

* Dealing with capstone

* Temporary disable ASR and remobe BitVec.Bool from test

* WIP WIP debug prints WIP WIP

* Unicorn fallback working (using unicorn master)

* HAck to support unicorn 1.0.0

* WIP

* Unicorn hack to handle PC updates

* [WIP] do not do anything with this commit; for debugging only

* Adding before clean up

* emulation more or less works; need to work out more unicorn bugs

* clean up emulate() caller code

* move hooks to methods; cleanup

* Concretize memory when emulating

* Re-add Bool()

* Update tests to start at offset 4

 When an instruction branches to the previous instruction,
Unicorn attempts to dereference that memory. We'd like to use
unit tests to also make sure Unicorn emulation is in line with
our own semantics. If we start all tests at offset 4, we can
jump to a previous instruction and not fault when Unicorn
dereferences it.

* Fix concretization

* Clean up test imports; upper-case Cpu

* Unicorn tests

 * Add tests for all the ARM semantics, but make sure they're equivalent
   on unicorn.
 * Add a few tests to make sure unicorn correctly concretizes the memory
   it references

* Fix broken import

* Add symbolic register tests

* Re-introduce the unicorn hack

* Add the 'ONE' concretization policy

* Rm unused function

* Update concretization; add comments

* Add ONE policy test

* Create a base class for all concretization exceptions

* Remove Armv7Cpu._concretize_registers

* Check for enabled logging in a more idiomatic way

* [wip] intermediate testing commit

* Reimplement hooks and execution with unicorn

* Add a DMB (mem barrier) instruction; nop

* simplify instruction resolution

* improve unicorn error handling

* explicitly delete emu

* Handle ARM helpers inline

* map fetched memory

* Narrow exception handling

* Update DMB docs; make __kuser_dmb match real implementation

* Fix typo; add comment; remove extraneous parameter

* typos++
2017-04-03 16:00:49 -04:00
Mark Mossberg
ffe758f5f3 Fix interpreter loader (#115)
* Correctly compute total interp size

* Rename

* Make staticmethod

* Add basic loading test
2017-03-31 13:22:59 -04:00
feliam
83cb4de825 Dev fix load interpreter (#96)
* Fix interpreter load

* turn off some debug

* cuack

* Correct loader condition (#111)

* Rm unnecessary condition

We always want to run this code if there is an interpreter present, not
just if the exe is ET_DYN

* Add correct interp header type check
2017-03-28 15:45:08 -03:00
feliam
dbb63cfa34 Dev stoping criteria (#108)
* no-mp not uset and redundant

* Add timeout

* Unused args reremoved timeout readded

* no-mp not uset and redundant

* Add timeout

* Unused args reremoved timeout readded

* Move import to the top op op op
2017-03-28 14:48:23 -03:00
feliam
6b1c69cb28 Dev fix decree (#98)
* Re enable skipped test

* Re enable skipped test

* Hack so we can use upstream Elftool

* pr comments

* pr comments - staticmethod

* pr comments - implicit > explicit
2017-03-24 19:10:33 -03:00
Yan
f0a6f66d77 Update MOV implementation (carry) (#105)
* Update MOV implementation wrt carry
* Remove intermediate flags dict
* Fix register reference
* Document MOV to conform with current standard
2017-03-24 17:55:24 -04:00
feliam
7dd0879918 Dev arm regfile refactor (#57)
* RegisterFile refactor everywhere

* Update arm_rf tests

* Refactor Operand in both x86 and arm

* Add APSR support (#53)

* Add APSR support

* #issuecomment-284826572

* APSR tests

* cspr to aspr

* RegisterFile refactor everywhere

* Update arm_rf tests

* Refactor Operand in both x86 and arm

* Rebased and NotImplementedError fix

* ARM register initialization compressed

* Fix merging bugs

* Assimilate PR comments

* Assimilate PR comments2

* The new style. future is now

* WIP

* Improve quick start ,add asciinema (#58)

* Towards a helloworld world

* Rm requirements.txt (#56)

* Print workspace dir by default (#55)

* Print workspace dir by default

* Change initial cli output

* refactor or or or

* WIP

* Add arm rf testcase

* ARM aliases fix

* debug print removed

* fix tests

* Remove unicorn script from travis build (#64)

* remove unicorn script from travis build

* remove unicorn script entirely

* Rename test -> tests (#66)

* rename test -> tests

* re-add ignored tests

* Switch instructions to prefer virtualenv (#65)

* switch instructions to prefer virtualenv

* document use of virtualenvwrapper

* Make cpuid more forgiving (#67)

* Make cpuid more forgiving

* error() to warning()

* Add CPUID EAX=0xd ECX=0,1

* Let logger handle the format string iff needed

* to hex

* Rm unused files (#76)

* Move state into own file (#75)

* Move State from executor.py

* Fix import

* relative import

* Rm unnecessary code (#80)

* Add example crackme (#77)

* add example crackme

* make filename and directory structure more accurate

* Fix state.branches (#74)

* init

* rm old code

pretty sure it's wrong

* Clean

* rm

* Rename record_fork, move to ConcretizeRegister exception handler

It doesn't need to be in fork() because it's only relevant for when
we fork due to PC.

* Add test for record_branches

* Add back record_branches which got removed in rebase

Rebase went oddly because State got moved to a new file

* unnecesary tuple()

* necesary tuple

* remove unnecessary keys()

* Use OrderedDict (#84)

It can be useful to know elements in `branches` towards the end are
more recently discovered branches

* arm: Fix broken dynamic APSR (#78)

* Refactor Register (#82)

* Fix register read bug

If we store a BitVec, we do NOT want to return bool() of it, which
returns True, which is totally incorrect. We do however, want to return
a symbolic Bool of it, if it is a 1 bit register (flag).

* Improve

* Raise error

* Fix register tests

Changed the interfaces:
- removed nbits param (never used)
- no longer raise AssertionError when overflowing a flag reg, just
  truncate
- rename test funcs to be more descriptive

* rm unused import

* Add symbolic tests

* Rm no longer applicable test

* Add docs (#61)

* Add sphinx-quickstart generated docs

* Add api.rst

* Edit index.rst

* Add autodocumenting Manticore class

* Update Manticore docstring

* Doc verbosity

* Doc hook and add_hook

* More docs

* Add Sphinx dev dep

For building docs

* RegisterFile refactor everywhere

* Update arm_rf tests

* Refactor Operand in both x86 and arm

* RegisterFile refactor everywhere

* Refactor Operand in both x86 and arm

* Rebased and NotImplementedError fix

* ARM register initialization compressed

* Fix merging bugs

* Assimilate PR comments

* Assimilate PR comments2

* The new style. future is now

* WIP

* Towards a helloworld world

* refactor or or or

* WIP

* Add arm rf testcase

* ARM aliases fix

* debug print removed

* fix tests

* unnecesary tuple()

* necesary tuple

* remove unnecessary keys()

* rebased

* Remove test

* https://github.com/trailofbits/manticore/pull/57#pullrequestreview-27971778

* https://github.com/trailofbits/manticore/pull/57#discussion_r107820815, https://github.com/trailofbits/manticore/pull/57#discussion_r107820331

* https://github.com/trailofbits/manticore/pull/57#discussion_r107821090

* https://github.com/trailofbits/manticore/pull/57#discussion_r107821066

* https://github.com/trailofbits/manticore/pull/57#discussion_r107821919

* OOps search and replace

* (invalid) -> None

* The (invalid) -> None

* None vs. invalid

* In armpy we know that STACK is SP so lets try to eliminate cpu.STACK in favor of x.SP

* remove ugly hex

* Removed redundant import and comment
2017-03-24 14:44:24 -03:00
feliam
f016b7635a Heads removed (#99) 2017-03-22 20:45:14 -03:00
Mark Mossberg
d0f2c0116e Refactor Register (#82)
* Fix register read bug

If we store a BitVec, we do NOT want to return bool() of it, which
returns True, which is totally incorrect. We do however, want to return
a symbolic Bool of it, if it is a 1 bit register (flag).

* Improve

* Raise error

* Fix register tests

Changed the interfaces:
- removed nbits param (never used)
- no longer raise AssertionError when overflowing a flag reg, just
  truncate
- rename test funcs to be more descriptive

* rm unused import

* Add symbolic tests

* Rm no longer applicable test
2017-03-21 16:50:39 -04:00
Mark Mossberg
5368716e42 Fix state.branches (#74)
* init

* rm old code

pretty sure it's wrong

* Clean

* rm

* Rename record_fork, move to ConcretizeRegister exception handler

It doesn't need to be in fork() because it's only relevant for when
we fork due to PC.

* Add test for record_branches

* Add back record_branches which got removed in rebase

Rebase went oddly because State got moved to a new file
2017-03-20 17:56:25 -04:00
JP Smith
ca0bee2377 Rename test -> tests (#66)
* rename test -> tests

* re-add ignored tests
2017-03-13 14:06:36 -05:00