Commit Graph

55 Commits

Author SHA1 Message Date
Theofilos Petsios 715cc436c7 Abstract Disassembler classes and Binja CPU + Platform (#364)
* adding abstract disasm class

* before adding abstract insn

* explicit capstone use

* (wip) removing capstone

* debugging nose

* removed disassembler from constructor

* nits

* capstone->cs and nits

* basic memory (wip)

* tmp

* added binja dissasm

* fixed execution pipeline - calling LLIL instructions

* updated parseargs to select disassembler

* unstable (disassembler at CPU constructor breaks tests)

* disasm fix for tests

* nit for aliases in x86

* added a flag to differentiate disassembly and IL-disassembly for binja

* before merging

* merging with dev-events master

* fixed signal forwarding for binja

* cleaner initialization

* Binja RegisterFile. Dropped Platform

* address fixup

* properly incrementing PC

* some work on XOR, SET_REG, REG, CONST, AND, PUSH, POP

* adding first CALL, JMP, LOAD (wip)

* JUMP fix

* GOTO and misc fixes in PC handling

* adding instructions (wip)

* fixing flags and sizes (wip)

* loading database if exists for faster analysis

* fix for register debug

* fixes for register sizes and flags

* FIX for multiple IL instructions sharing the same PC

* removed CONST_PTR and misc fixes

* RET, SHR, shift left & misc instructions, fixing insn sizes (wip)

* flag fixes

* ctypes 2's complement parsing

* JUMP_TO, SYSCALL

* syscall fixups and flag computation using binja il only

* FLAG, NEG, CMP(family) of LLIL, some unimplemented methods

* MUL, DIV

* cmpxchg

* MUL, IMUL, ROR, ROL

* fallback to capstone for all LLIL_UNIMPL and LLIL_UNIMPL_MEM

* fixes for registers and memory when switching CPUs

* check for binja to disable multiprocessing

* merging

* nit

* hack for serialization

* moved check for disassembler to Manticore from __main__

* removing __class__ refs

* cleanup from __class__.disasm

* size calculation from get_instruction_low_level_il

* fix for NEG, check for empty queue, execute refactor

* fixes for LLIL functions

* removal of redundant regfile writes

* nit fixes

* function overrides in Cpu classes and orphan printf cleanup

* nit

* incomplete merge

* verbosity temp mod

* cleanups of FIXMEs

* 2-stage constructor for disassembler

* cleanup binja refs in abstractcpu

* serialization for platform_cpu

* check for UNIMPL in all the il queue

* typo fix

* fix for arm

* typo correction and starting caching implementation

* restored register printing and fixed import in x86.py
2017-08-17 17:43:27 -04:00
Garret Reece 5d87fc83b3 Instructions and tests to support issue binary for issue #182 (#403)
* orn instruction plus unit test
* uadd8 and it instructions implemented
* SEL instruction implementation
* GE flag set by UADD8
* IT instruction condition code properly no longer prevents its execution
* support for multiple instruction tests added to testing setup
* unit test for SEL instruction
* cleaned up the tests for thumb instructions
* implemented sxth and rev instructions
* implemented tests for sxth and rev instructions
* updated the add implementation to support two operand variant (i.e., add r4, #4)
* added test for itete ne instruction
2017-08-17 10:02:44 -05:00
Theofilos Petsios 09bb702fac Implement logging presets (#445)
* logging template nits

* moved logging outside Manticore

* naming fix and exposed API call

* logging -> log renaming

* fix for verbosity test

* restored setter/getter for verbosity

* fixes for warnings in 0 mode and exteneed test_cli_verbosity

* removed print

* relative import, fork message to executor, init function

* verbosity docstring fix

* docstring nit
2017-08-14 16:05:29 -04:00
Mark Mossberg 1d8e051522 Fix state.generate_testcase (#451)
* Use correct event name

* Add test for state.generate_testcase

Test merely tests that this function publishes the 'will_generate_testcase' event

* Properly test; make sure the callback executes and error if not

This is the most convoluted unit test I've ever written.

* "better"
2017-08-14 14:44:53 -04:00
Mark Mossberg eef3cd20e2 Implement state.generate_testcase (#438)
* ran

* Some dirty work

* Rm unnecessary sanitization

* Fix state.generate_inputs event publishing

We don't need to manually pass `self` because State is automatically
forwarded as an argument. Also correctly specify a message for the state
and expose to the API

* Rename

* Update test for new save_testcase args, make test more robust

* Simplify arguments related to generate_testcase_callback by removing default arg. Add test for this behavior

* Minor test fixup

* Add official API docs

* Remove unnecessary new event type
2017-08-08 18:12:55 -04:00
feliam 3b57c0d502 Fix context serialization (#425)
* Fix context serialization

* Fix test models.  Can not set a state constraints

* typo

* A few typos (constraints setter) and use of public properties in internal methods

* State __init__ needs to initialize platform constraints

* Internal methods use internal properties

* test_record_branches removed
2017-08-01 14:08:54 -03:00
Eric Hennenfent 51837df98b Add Taint Parameters (#414)
* Add taint args to buffer creation

Allows the `new_symbolic_buffer` and `symbolicate_buffer` functions to take keyword args for tainting. Defaults to frozenset in both cases.

* Add unit tests and ArrayProxy taint propery

Adds simple unit tests for tainted buffers. Added a property to the ArrayProxy class in smtlib.expression so that it's possible to access the taint of the proxied ArrayVariable.

* Updated docstrings
2017-07-28 11:58:25 -04:00
feliam 76357216da Pub-sub like events (#371)
* WIP New Policy class

* WIP pubsub

* Update Signal tests

* small fixes from github comments

* Fix event decode_instruction signature

* Good merge

* Good good merge

* Eventful class commented

* The million typos

* Code revision

* Fix tests for new mor strict Eventful
2017-07-27 19:41:08 -03:00
JP Smith 13964d053b create workspace before running manticore to make memdump tests pass (#388) 2017-07-13 13:57:18 -04:00
JP Smith 60fa4c2231 Fix memdump tests broken by dev-events (#377)
* fix memdump tests broken by dev-events

* switch to superset check to account for timeout differences
2017-07-12 18:11:08 -04:00
Yan 45d5b9f6d2 New state serialization (#376)
* Fix naming

* Separate storage from workspace

* Begin removing output generation from manticore.py

* Split up workspace and output

* Create a separation between output and workspace

* Get it to a working (hacky) state

* Start bringing state serialization into workspace

* More mcore->workspace moves

* Remove unused imports

* Update serializers; add interface

* move state saving to workspace

* can now save/load states and testcases

* Add redis as backend

* Implement streams in terms of values and vice versa

* Implement `ls` and move workspace loading to Workspace

* Better workspace initialization

* Fix how --workspace is parsed for tests

* update tests to new workspace indexes

* Fix state storage and handling

* Doc updates and extra error checking

* pep8 style changes

* Add Executor's 'sync'

* Remove previous workspace artifacts from Executor

* Comment out constraints check assert

* Centralize locking to new testcase ids

* Fix testcase generation

* pep8 updates

* propagate descriptor instead of path

* Add an in-memory state store

* Make memory maps be sortable

* Allow to produce an empty state

* Add workspace tests

* Make Memory.__cmp__() more robust
2017-07-12 17:50:24 -04:00
JP Smith b6a71375cf Remove md5 tests and replace with set comparison (#374)
* remove md5 tests and replace with comparison

* fix directory path issue

* remove exact check as test is not deterministic

* fix argument assertion test and remove cdet_visited.txt

* allow extra addresses in actual to account for inter-machine differences
2017-07-07 15:43:52 -04:00
Mark Mossberg e4102e5000 Catch and refactor MemoryExceptions (#370)
* initial

* Clean, only store address in message if not symbolic

If it's symbolic,it renders like

    Generating testcase No. 2 - Invalid symbolic memory access (mode:r) <<manticore.core.smtlib.expression.BitVecAnd object at 0x7fd6dcd14410>>

which is not useful output to the user.

* Format as hex

* Fix tests

* Rm comments
2017-06-30 17:47:55 -04:00
Garret Reece 7d3d9a9ed1 lsl.w in thumb mode fixed, plus tests (#350)
- blx <Rm> swaps mode appropriately
- asr.w and lsr.w in thumb mode handle immediates for op2 correctly
- unit tests for thumb mode behavior for shifting instructions
- unit tests check for correct mode on BX and BLX instructions
2017-06-27 13:52:20 -05:00
Garret Reece e931992abb Add ldrd and strd instructions and tests (#348)
* LDRD and STRD implementation and tests
2017-06-26 17:22:59 -05:00
feliam 520a9be47d Dev - events (#341)
* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* forwarding events wip

* state setstate fix and setup_stack merge fix

* will_terminate_state fix and tests skipped

* Update all ConcretizeRegister and ConcretizeMemory

* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* Wip refactoring

* Executor and exceptions refactor wip wip

* Fixing all_insts auto tests

* Visited and generate testcase now at manticore api level

* Aggregating state statistics into executor statistics

* forwarding events wip

* state setstate fix and setup_stack merge fix

* will_terminate_state fix and tests skipped

* Update all ConcretizeRegister and ConcretizeMemory

* Exceptions are crazy crazy crazy

* fix last merge

* Merge merge until it pass

* Instructions count default to 0

* will/did execute/emulate

* Delayed keybpoard interrupt now shutdowns nicely

* fix auto test generator x86

* Undo bad merge

* utterly hopeless

* basic working

* Fix merge bugs and github comments

* Remove unnecesary comment - github comments

* trace_item not used there

* model-platform and system.py fixed

* backup/restore to store/load -- cpu.instruction property

* Slightly better did/will naming and dynamic signal forwarding

* platform.constraints and cpu.instruction as properties

* Fix forward signals getattr

* set las decoded pc at decode_instruction() / reenable instruction_cache

* Signals name convention: did/will/on

* Forward normal signals

* Maintain last decoded pc in abstractcpu

* Changed context manager so it just wont raise interrupt

* Decree now forwards signals and sets constraints

* linux.SymbolicFile does not need to maintain constraints

* remove debbug print

* Assimilating some PR commets

* size_total == size

* better merge of manticore.py

* typo

* Forwarding only specified objects in signal arguments

* Fix few broken tests

* revert + merge

* remove some unused stuff from manticore()

* manticore context <-> executor context

* manticore context <-> executor context2

* context context context

* forgotten return

* Fix basix.arm

* arm bitwise fix

* fix context

* Comment 1

* Comment 2

* Comment 3

* Comment 4

* Comment 5

* Comment 6

* Fix (still needs refactor but it works) profiling

* Fix (still needs refactor but it works) profiling

* The forgotten bit

* Update tests to reflect current output

* Verbosity fix

* Fix verbosity test
2017-06-26 18:06:18 -03:00
Theofilos Petsios e2c0414dca added _multiprocess_can_split_ directive (#351)
* added _multiprocess_can_split_ directive

* renamed Readme
2017-06-23 17:53:19 -04:00
Mark Mossberg 3ec9043b2e Fix verbosity=1 (#345)
* Only enable debug executor after verbosity 1

* Add regression test for checking uniform cli output for nonverbose mode
2017-06-21 17:48:37 -04:00
Garret Reece 7f550becc6 Add mrc and movt
* Add MOVT instruction

* Add new ARM operand types for MRC

* mrc implementation seems to work

* UXTB inst implemented, provisional implementations of LDREX and STREX

* Added unit tests for MRC, MOVT, and UXTB

* Added mode swapping from blx instruction on arm cpu

* Make tests pass
2017-06-13 13:21:19 -05:00
JP Smith 5ea4a87ddb Fix SymbolicMemoryException (#303)
* fix SymbolicMemoryException

* make simple_parse's visited.txt more correct
2017-06-12 15:28:57 -04:00
JP Smith 31b39ccf9d unskip win32 API test (#294) 2017-06-09 11:39:28 -04:00
JP Smith c7f5600cd1 unskip basic stdin test (#293) 2017-06-09 10:47:55 -04:00
Mark Mossberg 5b5392e54d Implement user variadic model interface (#276)
* Add variadic decorator

* Add check for _variadic func attribute

* Expose @variadic at top level

* Use variable for the actual name of attr

* Update naming

* Use helper

* Maybe more pythonic?

* Add variadic to public API

* Add variadic decorator tests

* Clean

* Remove variadic parameter

* Add docstrings

* Document parameter

* Clean

* Better use of autodoc

* Update docstring

* Add todo comment
2017-05-31 15:48:19 -04:00
Mark Mossberg 8354ded7e0 Test hook api (#274) 2017-05-25 16:22:08 -04:00
Mark Mossberg c2b0617f32 Check returncode of manticore (#273) 2017-05-25 16:04:34 -04:00
Yan 3c977719f0 Linux platform refactoring (#264)
* Clean up model syscall invocation
* Move read_string to Cpu
* move push/pop helpers to Cpu
* Reorg Linux initialization
* Update linux test
* fstat64 test harness
* assert read_string only considers concrete bytes
* Ensure that correct aliases exist during ctor
* Improve alias check
2017-05-23 15:30:58 -04:00
feliam 475f786f21 Fix x86.PUNPCKL emulation (#259)
* Fix x86.PUNPCKL emulation

* trace check wip

* PUNPCKH/L and Readlink /proc

* PUNPCK fixes

* remove debug traceback

* rename arguments insteag of commenting them

* fix comment
2017-05-23 15:21:48 -03:00
Mark Mossberg 8840e31fc2 Add strlen model (#260)
* add models

* wip strcmp tests

* t

* Add some tests

* Better asserts

* More pythonic

* Add effectivene null test

* init strlen

* pythonic range

* clean

* docstring

* tmp

* stuff

* stuff

* Complete test

* Add another test

* Doc

* clean

* Add symbolic arg check

* Fix tests

* Make arch generic

* Clean comment

* Add basic unconstrained test
2017-05-18 16:59:44 -04:00
Mark Mossberg 92eaf76236 Add strcmp model (#251)
* add models

* wip strcmp tests

* t

* Add some tests

* Better asserts

* More pythonic

* Add effectivene null test

* Handle symbolic pointer arguments
2017-05-17 11:44:10 -04:00
Mark Mossberg 7700112d9e Rm unused Slinux ctor params (#250)
* Rm unused Slinux ctor params

Fixes #109

* Add back constraitns

* Make argv and envp default params

* Use ternary

* nit

* Clean test_unicorn

* Correct envp and docstring
2017-05-12 11:35:59 -04:00
Mark Mossberg e4a4916597 Rename os model terminology from "models" to "platforms" (#243)
* Rename

* rename in manticore.py

* rename in executor.py

* big rename

* big rename

* update changelog
2017-05-09 19:25:32 -04:00
Yan 3c9653d1d7 Update syscall/funcall implementation (#213)
* Wrap ABI semantics in its own class hierarchy

* Define a model invocation for syscalls and function calls

* Add unit tests for ABI

* Add a common base class for Platform models
2017-05-09 16:13:08 -04:00
Yan 395a40a646 Add memory tracing (#203)
* implement memory write tracing

* Comment updates

* Add memory trace tests

* make sure we ignore erroring writes

* Address comments

* remove superfluous return annotation
2017-05-08 14:32:54 -04:00
Mark Mossberg 44d365ff4c Formalize API for constraining a State (#232)
* Rename state.add to state.constrain

* Update all uses of state.constrain

* Rm check param

* Added changelog

* Update changelog

* Update

* Minor clean
2017-05-05 13:31:43 -04:00
Yan 50fd50ee12 Simplify linux model interfaces (#233) 2017-05-05 13:11:59 -04:00
Yan 64d1ed51f2 Decode instruction early (#205)
* Move PC verification to decode_instruction

* Decode instruction in executor

* Fix unicorn tests

* remove decoded_pc

* use issymbolic()
2017-05-04 17:36:39 -04:00
Yan 8d6bcadb37 Fix RSB, VLDMIA; Add RSC (#197)
* Fix RSB; Add RSC
* Fix vldmia semantics
* To trigger build
2017-04-27 16:36:23 -04:00
Mark Mossberg fd3873f0a8 Fix ARM bitwise (LSL/LSR) (#174)
* Rm bogus asserts

* correct lsl_c

* Add test for lsl carry
2017-04-24 11:31:11 -04:00
Mark Mossberg a52be7459d Arm add vldmia (#154)
* Add VLDMIA

* Add vldmia tests
2017-04-20 15:30:50 -04:00
Mark Mossberg 3e81cba8fb Fix linter issues on cpu (#144)
* quick linter stuff

* Fix cpu init in tests

* update cpu name
2017-04-19 10:29:05 -04:00
JP Smith e32701f978 Refactor logging (#140)
* begin refactoring logging to use manticore object

* fix verbosity things

* fix logging when in scripts

* remove m.log_file now that it's no longer used

* remove unnecessary init_logging usage and unused log_debug variable

* accidentally deleted the wrong line

* re-hide init_logging

* remove old log_debug

* remove unnecessary API stuff and refactor around that

* re-introduce logging PR

* fix Popen

* lost a line in merge

* fix test_binaries

* Log to stdout instead of stderr

* implement mark's changes
2017-04-19 01:21:14 -05:00
feliam a9711cf119 Graceful fallback if Z3 doesn't support optimization queries (#135)
* Configure Z3 depending on installed version

* Mocking for subprocess.check_output

* change get_value_fmt to private attribute _get_value_fmt

* Move memoized to utils

* Remove minimal version and simplify

* Re-add invalid versions/sanity checks

* Z3 version format checked on github

* New tests. Version is future proof

* Better logging
2017-04-18 18:40:06 -03:00
Mark Mossberg 05b7c1138b Fix memory._ceil, Fix linux.load (#134)
* Reimplement ceil so page aligned addresses are unaffected

* Lingering fix

* Fixup users of _ceil

* Correct memory tests
2017-04-13 18:45:18 -04:00
feliam bbfb234cca Dev fix amd64 emulation bugs (#136)
* Failing tests

* Tha Fix

* Fix auto test

* PSLLDQ for 32b python?

* More descriptive temporary variable
2017-04-13 19:22:04 -03:00
Yan 0b850dba40 Make unicorn be pull-based (#97)
* Enable simple ARM register concretization for Unicorn

* Add canonical_registers property to abstractcpu

* cpu to self

* Check for regs_access better

* Emulate a single instruction

* Bypass capstone 3.0.4 arm bug

* Dealing with capstone

* Temporary disable ASR and remobe BitVec.Bool from test

* WIP WIP debug prints WIP WIP

* Unicorn fallback working (using unicorn master)

* HAck to support unicorn 1.0.0

* WIP

* Unicorn hack to handle PC updates

* [WIP] do not do anything with this commit; for debugging only

* Adding before clean up

* emulation more or less works; need to work out more unicorn bugs

* clean up emulate() caller code

* move hooks to methods; cleanup

* Concretize memory when emulating

* Re-add Bool()

* Update tests to start at offset 4

 When an instruction branches to the previous instruction,
Unicorn attempts to dereference that memory. We'd like to use
unit tests to also make sure Unicorn emulation is in line with
our own semantics. If we start all tests at offset 4, we can
jump to a previous instruction and not fault when Unicorn
dereferences it.

* Fix concretization

* Clean up test imports; upper-case Cpu

* Unicorn tests

 * Add tests for all the ARM semantics, but make sure they're equivalent
   on unicorn.
 * Add a few tests to make sure unicorn correctly concretizes the memory
   it references

* Fix broken import

* Add symbolic register tests

* Re-introduce the unicorn hack

* Add the 'ONE' concretization policy

* Rm unused function

* Update concretization; add comments

* Add ONE policy test

* Create a base class for all concretization exceptions

* Remove Armv7Cpu._concretize_registers

* Check for enabled logging in a more idiomatic way

* [wip] intermediate testing commit

* Reimplement hooks and execution with unicorn

* Add a DMB (mem barrier) instruction; nop

* simplify instruction resolution

* improve unicorn error handling

* explicitly delete emu

* Handle ARM helpers inline

* map fetched memory

* Narrow exception handling

* Update DMB docs; make __kuser_dmb match real implementation

* Fix typo; add comment; remove extraneous parameter

* typos++
2017-04-03 16:00:49 -04:00
Mark Mossberg ffe758f5f3 Fix interpreter loader (#115)
* Correctly compute total interp size

* Rename

* Make staticmethod

* Add basic loading test
2017-03-31 13:22:59 -04:00
feliam 83cb4de825 Dev fix load interpreter (#96)
* Fix interpreter load

* turn off some debug

* cuack

* Correct loader condition (#111)

* Rm unnecessary condition

We always want to run this code if there is an interpreter present, not
just if the exe is ET_DYN

* Add correct interp header type check
2017-03-28 15:45:08 -03:00
feliam dbb63cfa34 Dev stoping criteria (#108)
* no-mp not uset and redundant

* Add timeout

* Unused args reremoved timeout readded

* no-mp not uset and redundant

* Add timeout

* Unused args reremoved timeout readded

* Move import to the top op op op
2017-03-28 14:48:23 -03:00
feliam 6b1c69cb28 Dev fix decree (#98)
* Re enable skipped test

* Re enable skipped test

* Hack so we can use upstream Elftool

* pr comments

* pr comments - staticmethod

* pr comments - implicit > explicit
2017-03-24 19:10:33 -03:00
Yan f0a6f66d77 Update MOV implementation (carry) (#105)
* Update MOV implementation wrt carry
* Remove intermediate flags dict
* Fix register reference
* Document MOV to conform with current standard
2017-03-24 17:55:24 -04:00