Python Virtual Environments in Windows, virtualenv, lsvirtualenv, venv

sSome commands for Anaconda:
  • Which version do I have?
  • > conda --version
  • The latter actually recommended me to update the conda using conda update -n base conda, so…
  • > conda update -n base conda
  • Creating a new venv: Following, I ran:
  • > conda create -n venv_tf python=3.5
    (venv_tf) > pip install --ignore-installed --upgrade tensorflow
  • Can then simply activate it from ordinary command prompt:
  • > activate venv_tf
    # Deactivate the current venv
    > deactivate
    # Which environments I have?
    > conda env list
    # Get help:
    > conda env
    > conda env --help
  • Can then share your environment’s environment.yml file. See
  • More useful commands from command line: (not necessarily related to Anaconda…)
    • lsvirtualenv to view virtual environments (assuming they were created by venv, and not by conda, because each of these maintains their venvs in different places, I guess).
    • To activate the virtual environment located in C:\Dev\venv\, simply run: C:\Dev\venv\Scripts\activate
    Other notes on PyCharm:
    • PyCharm’s built-in Terminal does NOT work as an external command line. I witnessed different error messages (formatted differently, less detailed) in the built-in Terminal. In one case, python did NOT work from the built-in Terminal, while the same command worked from a normal command line.
    • If the packages you install need for some reason Visual Studio C++ of some version, it did NOT help me to add to the PATH the directories where my Visual Studio C++ was located (namely, set PATH=C:\Users\%USERNAME%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\amd64;C:\Users\%USERNAME%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\WinSDK\Bin\x64;C:\Users\%USERNAME%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\WinSDK\Bin;%PATH%). Instead, press “Start”, then search for “Visual C++ Command Prompt” or something like that, and this command prompt already includes in its %PATH% variable the relevant directories.
    Other notes on python:
    • — Unofficial Windows Binaries for Python Extension Packages

    TensorFlow Cheatsheet

    General cheatsheet

    Initialize a Variable using constants or random values (zeros, zeros_like, linspace, etc.).

    Can save and restore models using tf.train.Saver.

    Read on from here for more cool stuff:

    Graph vs. Session

    (following the great explanation by Danijar Hafner)
    graph = tf.Graph()
    with graph.as_default():
        v = tf.Variable(tf.random_normal(shape=[1]), name='foo')
        print v.shape, v.shape.ndims, v.shape.num_elements()  # (1,) 1 1
        if 1: # Don't do this! tf.global_variables_initializer() defines an op that initializes all variables in the graph (so far); so you should call this AFTER they were all defined, otherwise you'll get something like "FailedPreconditionError: Attempting to use uninitialized value FailedPreconditionError: Attempting to use uninitialized value foo_scalar"
            init_op_notgood = tf.global_variables_initializer()
        v_scalar = tf.Variable(tf.random_normal(shape=[]), name='foo_scalar'))  # shape=[] or shape=() defines a 0-dimensional tensor, i.e. scalar
        print v_scalar.shape, v_scalar.shape.ndims, v_scalar.shape.num_elements()  # () 0 1
        init_op = tf.global_variables_initializer()  # Add an op to initialize all variables in the graph (actually probably best to define this even further down after the entire graph was constructed, but defining it here is already okay for our example)
        assign_v = v.assign([101])
        assign_v_scalar = v_scalar.assign(102)
    c = tf.constant(4.0)  # Will be defined as attached to the default graph (!) tf.get_default_graph()
    # Sanity check
    print c.graph == tf.get_default_graph(), variable.graph == graph, graph == tf.get_default_graph()  # True True False

    Then instantiate a Session to run our graph:

    with tf.Session(graph=graph) as sess:
        print  # e.g. [-0.407900009]
        print  # e.g. 1.30248
        if 1:  # Don't do this part
            print  # e.g. [0.33414543], a different value than above
            print  # e.g. 1.30248, same as above
        print[assign_v, assign_v_scalar])  # [array([ 101.], dtype=float32), 102.0] -- return values probably not really interesting here
        print  # [ 101.]
        print  # 102.0
        if 0: # Error, as 'c' is not an element of the current graph...
            print  # ValueError: Fetch argument <tf.Tensor 'Const_50:0' shape=() dtype=float32> cannot be interpreted as a Tensor. (Tensor Tensor("Const_50:0", shape=(), dtype=float32) is not an element of this graph.)

    Docker in Windows, allocate more memory

    Some installation / cheatsheet

    Using Docker for Windows, Windows 10. 17.03.0-ce-win1 (10296). Channel: stable. 94675c5.

    (Tip: Also install Kitematic, then you can right-click the running Docker icon and from there launch Kitematic for a nice GUI to see your containers.)

    Basic commands:

    # Create a tensorflow container:
    docker run -it -p 8888:8888 -p 6006:6006 --name my_tensor_flow -v C:/Data/Docker:/data -v C:/Dev/python:/devpython tensorflow/tensorflow
    # Run it in the future:
    docker start -ai my_tensor_flow
    # To connect to it with bash: (note: /notebooks is where the notebooks are kept, and we can already access the host's C:/Data/Docker in /data)
    docker exec -it my_tensor_flow bash
    # Then from the bash can run: tensorboard --logdir=/data/tensorboard/5/

    Apart from the nice GUI, there are some useful commands from the command line:

    docker stats [--all]
    docker ps [--all]
    docker container list [--all]

    Allocating more memory

    For my running container, docker stats showed (in “MEM USAGE / LIMIT”) that it was bounded by 1.934 GiB. Indeed, “Hyper-V Manager” showed “MobyLinuxVM” machine has only 2GB.

    To allocate more memory: Docker’s settings, Advanced, set Docker’s memory e.g. to 8448MB. Docker will restart. Verify that “Hyper-V Manager” now shows “MobyLinuxVM” has assigned memory of 8448 MB. Run your previous container docker start -ai container-name, and now docker stats will show a memory bound of 8.003 GiB. Success.

    Note: Running some

    docker run -it -p ... --name ... --memory-swap -1 --memory 8g -v ...
    will not work if your “MobyLinuxVM” doesn’t have enough assigned memory.

    Git Quick Tutorial, Cheat Sheet

    The following is meant to be a quick cheatsheet to recap some Git commands.

    Basic flow

    An example basic flow of commands:
    git initto create an empty Git repository in a .git folder.
    git statusto see which files are not tracked, changes, conflicts etc.
    git add .to add all files to the staging area (a.k.a. “index”).
    (if you modify a file now and want to commit those changes, don’t forget to git add it again before committing)
    git reset file.txtto undo that last add command for file.txt, so now it’s not tracked.
    git commit -m "my commit remark" creates a new commit object with the staged changes in the Git repository.

    Deleting files

    Create some file and commit it: touch deleteme.txt git add . git commit -m "added deleteme.txt" Will commit the deleteme.txt file
    Try to delete (will FAIL): rm deleteme.txt git add . git commit -m "will fail" Will get a no changes to commit message, and the deleteme.txt file will still be kept in the repository…
    How to delete (alternative 1): git add -A . git commit -m "success" The -A (or equivalently --all) compares changes in the working tree as well as the index, so the file deleteme.txt will now be deleted in the repository as well.
    How to delete (alternaive 2): git commit -a -m "success" The -a (or equivalently --all) tells the commit command to automatically stage files that have been modified and deleted, so the file deleteme.txt will now be deleted in the repository as well.
    How to delete (alternaive 3): Instead of the rm deleteme.txt above, could instead: git rm deleteme.txt git commit -m "success" The rm command removes the file and records its deletion in Git, so after the commit command the deleteme.txt will be deleted in the repository as well.


    git diff shows the changes in the working tree not yet staged for the next commit. git diff --cached shows the changes between the index and your last commit; what you would be committing if you run “git commit” without “-a” option. In any case, it ignores any non-tracked files.
    echo "some text" > file.txt git add file.txt git commit -m "remark1" Suppose we add a file.txt and commit it.
    Now modify it: echo "more text" >> file.txt Modified, but still didn’t add it to staging, i.e. we didn’t run: git add file.txt
    git diff will show us the differences we have in that tracked file! Example diff output: diff --git a/file.txt b/file.txt index 477f637..f8217a9 100644 --- a/file.txt +++ b/file.txt @@ -1 +1,2 @@ "some text" +"more text"
    But after adding the changes there will be no diff: git add file.txt git diff So the git diff will not print anything now. If you want to see the changes between the index and the last commit, use git diff --cached (will produce the same diff output as above)

    Remotes and more

    git remote List the existing remotes whose branches you track.
    git remote -v A bit more verbose, show remote url after name.
    Typical scenarios:
    • Create a new repository:
      echo "# My Space" >>
      git init
      git add
      git commit -m "first commit"
      git remote add origin path@of.origin:etc
      git push -u origin master
    • Push to an existing repository:
      echo "# My Space" >>
      git remote add origin
      git push -u origin master
    • Push --force to forcefully overwrite remote’s content with local’s. See StackOverflow questions here,here,here,…
    More stuff, possibly repeating myself:
    • Clear all history (!): remove the .git folder, then push --force etc., see gist
    • Add a remote repository: Use ‘git remote add’ with some remote name (usually “origin” is the name of the main remote) and repository URL:
    • git remote add origin
    • See which remotes you’re currently connected to: git remote
    • Remove remote origin: git remote remove origin. But instead can change its URL: git remote set-url origin git:// (see SO).
    • Now can push local changes remotely to our ‘origin’ repo (on GitHub): The name of our remote is ‘origin’. The default branch name is ‘master’.
    • git push -u origin master (the -u tells Git to remember the parameters, so that next time we can simply run git push and Git will know what to do)
    • Suppose others worked on our project and committed stuff. We can check for changes on our GitHub repository and pull down any new changes by running:
    • git pull origin master
    • If we want the diff of our most recent commit, which we can refer to using the HEAD pointer:
    • git diff HEAD
    • Another great use for diff is looking at changes within files that have already been staged/cached (added, but not committed yet):
    • git diff --cached (--cached is same as --staged)
    • Other example for git diff:
    • git diff v2.5:Makefile
    • To unstage files:
    • git reset octofamily/octodog.txt
    • To revert the working tree to the state of some branch or specific paths:
    • git checkout master (checks out the master branch (switches branch)) git checkout master~2 Makefile (takes a file out of another commit. Reverts the Makefile to two revisions back)
    • Now suppose we accidentally ran ‘rm -f hello.c‘. Then we could restore it from the index:
    • git checkout hello.c (but in case we have a branch called hello.c, this will be understood as reverting to that branch!!! So better to use the following:) git checkout -- hello.c
    • Branching out:
    • git branch my_branch
    • To see the branches: (with “*” at the current branch)
    • git branch
    • To switch to another branch: (also discussed above)
    • git checkout my_branch
    • In that branch, suppose we remove some files from both the disk and from the staging/index:
    • git rm '*.txt'
    • And commit the change:
    • git commit -m "cleaned up stuff"
    • Now we want to copy (merge) changes from my_branch back into the master branch:
    • git checkout master
      git merge my_branch
    • To delete a branch:
    • git branch -d my_branch
    • Finally, can again push it to the remote repo:
    • git push
    git log Shows the recent commits etc.

    Other resources