Python

This is a list of useful commands related to Python.

Jupyter notebook location

When starting jupyter notebooks on Windows the shortcut defaults to my profile directory.  To get round this create a jupyter config:

jupyter notebook --generate-config

This will be located in .jupyter\jupyter_notebook_config.py within the profile directory.

Update c.FileContentsManager.root_dir to point to the folder for notebooks

Machine learning

Frequency analysis with Python https://sites.google.com/site/haskell102/home/frequency-analysis-of-audio-file-with-python-numpy-scipy

scikit-learn documentation http://scikit-learn.org/stable/user_guide.html

Azure Machine Learning documentation

Sequential data

http://web.engr.oregonstate.edu/~tgd/publications/mlsd-ssspr.pdf

Fraud detection with Azure Stream Analytics https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-real-time-fraud-detection/

Streaming analytics in Python http://matthewrocklin.com/blog/work/2014/07/04/Streaming-Analytics

Anomaly detection

Azure anomaly detection

Azure One-Class Support Vector Machine

Enable RDS RestrictedAdmin mode

Enable for incoming connections:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa
Name: DisableRestrictedAdmin
Type: REG_DWORD
Value: 0

To require for outgoing connections:

  1. Edit the Group Policy and navigate to the following node:
    Computer Configuration\Policies\Administrative Templates\System\Credentials Delegation
  2. Configure the value of “Restrict delegation of credentials to remote servers” to Enabled.

See https://blogs.technet.microsoft.com/srd/2014/06/05/an-overview-of-kb2871997/ for additional security features.

E-mail security

Validation tools:

Configuration:

F5 BIG-IP Rest API Security

I’ve been using the BIG-IP rest API to read the configuration of LTM, and I wanted to reduce the risk of accidental changes.  During my investigation I can across the post: https://devcentral.f5.com/questions/read-only-access-to-icontrol-rest-api, which showed the way to configure role based access control for the Rest API.

I wanted to create a new rule for a read only account (‘monitor’), the account is configured as Guest within BIG-IP.  In order to do this I created a new access rule you need the POST method:

curl -k -u admin:adminpass -X POST https://1.1.1.1/mgmt/shared/authz/roles -d @addRule.json

where addRule.json is:

{"name": "iControl_REST_API_monitor","userReferences":[{"link":"https://localhost/mgmt/shared/authz/users/monitor"}],"resources":[{"resourceMask":"/mgmt/tm/ltm","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*/*/*","restMethod":"GET"}]}

This worked fine until I discovered I did not have enough depth in rules to cover the data I wanted to retrieve. To update the rule you need the PUT method:

curl -k -u admin:adminpass -X PUT https://1.1.1.1/mgmt/shared/authz/roles/iControl_REST_API_monitor -d @updateRule.json

where updateRule.json is:

{"userReferences":[{"link":"https://localhost/mgmt/shared/authz/users/monitor"}],"resources":[{"resourceMask":"/mgmt/tm/ltm","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*/*/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*/*/*/*","restMethod":"GET"},{"resourceMask":"/mgmt/tm/ltm/*/*/*/*/*/*","restMethod":"GET"}]}

TMG from PowerShell

I recently had to perform an audit of TMG rules and used PowerShell to access the object model.  Below are a couple of sample commands that show accessing various parts of the TMG object model with PowerShell.

$tmg = New-Object -ComObject "FPC.Root"
$proxy = $tmg.Arrays | ? {$_.DNSName -eq "PROXY"}
($proxy.ArrayPolicy.PolicyRules | ? {$_.Name -eq "Allow Sample Sites Access"}).AccessProperties.DestinationDomainNameSets
($proxy.ArrayPolicy.PolicyRules | ? {$_.Name -eq "Allow Sample Sites Access"}).AccessProperties.UserSets
$proxy.RuleElements.DomainNameSets.Item("Sample Sites")
$proxy.RuleElements.UserSets.Item("Sample Site Access Group").Accounts

The following links are useful reference material for interpreting TMG logs:

Encoding

Sample encodings taken from https://www.redspin.com/blog/labs/2009/07/24/string-encoding-in-the-shell/

#!/bin/bash
 
if [ $# -ne 1 ]
then
  echo "Performs a number of encodings on the first argument string"
  echo "Usage: `basename $0` {string}"
  exit 1
fi
 
printf "n# String Scrambles:n"
printf "%-20st" 'Normal:'; echo "$1"
printf "%-20st" 'Reversed:'; echo "$1" | rev
printf "%-20st" 'Case Reversed:'; echo "$1" | tr '[A-Z][a-z]' '[a-z][A-Z]'
printf "%-20st" 'ROT13:'; echo "$1" | gcipher -c Rot -k 13
#printf "%-20st" 'Rot13:' ; python -c "print '''$1'''.encode('rot13')"
printf "%-20st" 'GIE:'; echo "$1" | gcipher -c Gie
printf "%-20st" 'Caesar:'; echo "$1" | gcipher -c Ceasar
printf "%-20st" 'Vigenere:'; echo "$1" | gcipher -c Vigenere -k vigenere
# printf "%-20st" 'Anagrams:'; wordplay -s "$1" | sort -u | sed -n '1h;2,$H;${g;s/n/, /g;p}'
# Due to both terminal and editor encodings, this is better executed on a non-UTF8 terminal:
printf "%-20st" 'Leet (l334):'; echo "$1" | tr [a-z] [A-Z] | tr 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' '4ß(Ð3ƒ9H1JK£MN0PQ®$7µVWX¥2' | sed 's_H_|-|_g;s_J__|_g;s_K_|{_g;s_M_|\/|_g;s_N_|\|_g;s_P_|°_g;s_Q_¶¸_g;s_V_\/_g;s_W_\/\/_g;s_X_)(_g' #See http://www.albinoblacksheep.com/text/leet
 
printf "n# Numerical Representations:n"
printf "%-20st" 'INT:'; echo -n "$1" | hexdump -ve '/1 "%03i"'; echo
printf "%-20st" 'HEX:'; echo -n "$1" | hexdump -ve '/1 "%02x"'; echo
printf "%-20st" 'OCT:'; echo -n "$1" | hexdump -ve '/1 "%02o"'; echo
printf "%-20st" 'BIN:'; echo -n "$1" | xxd -b -g0 -c0 | cut -b10-56 | tr -d 'n '; echo
 
printf "n# Passwords:n"
printf "%-20st" "CRYPT w/o SALT:"; echo -n "$1" | openssl passwd -crypt -stdin -salt 00
printf "%-20st" "CRYPT w/ Random SALT:"; echo -n "$1" | openssl passwd -crypt -stdin
printf "%-20st" "DES w/ CR SALT:"; echo -n "$1" | openssl passwd -crypt -stdin -salt CR
printf "%-20st" "Shadow w/o SALT:"; echo -n "$1" | openssl passwd -1 -stdin -salt 00000000
printf "%-20st" "Shadow w/ RANDOM SALT:"; echo -n "$1" | openssl passwd -1 -stdin
printf "%-20st" "Apache w/o SALT:"; echo -n "$1" |  openssl passwd -apr1 -stdin -salt 00000000
printf "%-20st" "Apache w/ RANDOM SALT:"; echo -n "$1" |  openssl passwd -apr1 -stdin
printf "%-20st" "LM Password:"; python -c "import smbpasswd; print smbpasswd.lmhash("""$1""")" #requires python-smbpasswd
printf "%-20st" "NTLM Password:"; python -c "import smbpasswd; print smbpasswd.nthash("""$1""")" #requires python-smbpasswd
 
printf "n# Digest Hashes (newline not included):n"
#printf "%-20st" 'BINARY MD5:' ; echo -n $1 | openssl dgst -binary
printf "%-20st" 'MD5:'; echo -n $1 | openssl dgst -md5
printf "%-20st" 'MD4:'; echo -n $1 | openssl dgst -md4
printf "%-20st" 'MD2:'; echo -n $1 | openssl dgst -md2
printf "%-20st" 'SHA1:'; echo -n $1 | openssl dgst -sha1
printf "%-20st" 'SHA:'; echo -n $1 | openssl dgst -sha
printf "%-20st" 'SHA224:'; echo -n $1 | openssl dgst -sha224
printf "%-20st" 'SHA256:'; echo -n $1 | openssl dgst -sha256
printf "%-20st" 'SHA384:'; echo -n $1 | openssl dgst -sha384
printf "%-20st" 'SHA512:'; echo -n $1 | openssl dgst -sha512
#printf "%-20st" 'MDC2:' ; echo -n $1 | openssl dgst -mdc2
printf "%-20st" 'RIPEMD160:'; echo -n $1 | openssl dgst -ripemd160
printf "%-20st" 'CRC32:'; python -c "import binascii; print binascii.crc32('''$1''') & 0xffffffff" 
 
printf "n# Web Encodingsn"
printf "%-20st" 'URLQuote:'; python -c "import urllib; print urllib.quote('''$1''')"
printf "%-20st" 'URLEscape:'; echo "$1" | recode ..HTML
printf "%-20st" 'HTML HEX Entity:'; echo -n "$1" | hexdump -ve '/1 "&#x%02x;"'; echo
printf "%-20st" 'HTML Entity:'; echo -n "$1" | hexdump -ve '/1 "&#%02i;"'; echo
printf "%-20st" 'Javascript String'; echo -n "String.fromCharCode("; echo -n "$1" | hexdump -ve '/1 "%i,"' | sed 's_,$_)n_'
printf "%-20st" 'SQL String'; echo -n $1 | hexdump -ve '/1 "char(%i)+"' | sed 's_+$_n_g'
 
printf "n# UTF Encodingsn"
printf "%-20st" 'UTF-7:'; echo $1 | iconv -t utf7
printf "%-20st" 'UTF-8:'; echo $1 | iconv -t utf8
printf "%-20st" 'UTF-16:'; echo $1 | iconv -t utf16
printf "%-20st" 'UTF-32:'; echo $1 | iconv -t utf32
printf "%-20st" 'Unicode:'; echo $1 | iconv -t unicode
printf "%-20st" 'ASCII:'; echo $1 | iconv -t ascii
 
printf "n# Encodingsn" #http://docs.python.org/library/codecs.html#standard-encodings
printf "%-20st" 'Base64:'; echo -n $1 | openssl enc -e -base64
#printf "%-20st" 'Base64:'; python -c "import base64; print base64.b64encode('''$1''')"
printf "%-20st" 'Base32:'; python -c "import base64; print base64.b32encode('''$1''')"
printf "%-20st" 'Base16:'; python -c "import base64; print base64.b16encode('''$1''')"
#printf "%-20st" 'UUEncode:'; python -c "print repr('''$1'''.encode('uu_codec'))"
#printf "%-20st" 'UUEncode:';; echo -n $1 | hexdump -ve '/1 "#%02x"' | tr '#' '%'
printf "%-20st" 'UUEncode:'; python -c "import binascii; print binascii.b2a_uu('''$1''')" | tr -s 'n'
printf "%-20st" 'Punycode:' ; python -c "print '''$1'''.encode('punycode')"
printf "%-20st" 'Mime Quotable:' ; python -c "print '''$1'''.encode('quopri_codec')"
 
printf "n# Compression Encodingsn"
#printf "%-20st" 'Bzip2:' ; python -c "print repr('''$1'''.encode('bz2_codec'))" | sed "s_^'(.*)'$_1_"
#printf "%-20st" 'Zlib (gzip):' ; python -c "print repr('''$1'''.encode('zlib_codec'))" | sed "s_^'(.*)'$_1_"
printf "%-20st" '7z:' ; echo -n "$1" | 7z a dummy -tgzip -si -so 2>/dev/null | hexdump -ve '/1 "%02x"'| sed "s_(..)_\x1_g"; echo
printf "%-20st" 'Bzip2:' ; echo -n "$1" | bzip2 -f | hexdump -ve '/1 "%02x"'| sed "s_(..)_\x1_g"; echo
printf "%-20st" 'GZip:' ; echo -n "$1" | gzip -f | hexdump -ve '/1 "%02x"'| sed "s_(..)_\x1_g"; echo
printf "%-20st" 'Zip:' ; echo -n "$1" | zip 2>/dev/null | hexdump -ve '/1 "%02x"'| sed "s_(..)_\x1_g"; echo
 
#printf "n# OpenSSL Ciphers with empty passphrase, key and iv:n"
#for line in `openssl enc -h 2>&1 | sed -n '/Cipher Types/,//p' | grep -v -e "Cipher Types" -e "^$" | tr -s [:space:] 'n'`; do printf "%-20st" "$line:"; echo -n $1 | openssl enc -k "" -e -a -p -K 0 -iv 0 "$line" | sed -n '1h;2,$H;${g;s/n/, /g;p}'; done
 
#printf "n# All iconv Output Encodings ~= 1153:n"
#for line in `iconv -l`; do printf "%-20st" "$line"; echo -n $1 | iconv -t "$line" 2>/dev/null; echo; done

Python

import binascii
filename = 'test.dat'
with open(filename, 'rb') as f:
    content = f.read()
print(binascii.hexlify(content))

Cipher

Cipher tools http://rumkin.com/tools/cipher/