Started the day off skiing the first good snow we have had this year with Joey. A gentle white out and lots of soft, lumpy snow with bits of untouched powder to be found. I definitely am not as smooth in the soft snow as as I would like yet, but it was fun. Fatter skies would help.
Wrapping up release activities, which while tedious took Liz and I a lot less time than it has historically. Not much other than responding to forum posts and some reading:
I am trying wo
There were essentially just 2 things that happened today:
DCE has thus far been a very straight forward look at how computers operate at the level of the metal. Ch 7 was on flip-flops, which while not the most interesting chapter is a crucial foundational piece for being able to store information. Working through this book really has improved my mental model of how a computer fundementally operates.
Considering that Matt is now on his sabattical, this release has gone pretty smoothly. For the most part it feels like Liz and I are able to do most things and are pretty quick to figure out when we do not know. It is nice to be able to fail fast.
Ok, the subsampling script that I had been plannin on working on yesterday did not really go anywhere then, though I did get the q2-dada2 expected data output setup. So here we go on the subsampling stuff. Here are some design points:
Inputs: a manifest of the fastq files and the desired number of reads from each.
create directory to store subsampled files.
read files from manifest and subsample them one at a time.
If any of the fastqs are unzipped, unzip them(maybe un-necessary if sk-bio
can read compressed files.
Open each file using skbio.io.read
reservoir sample each file to the target number of reads. This function should be a generator that takes the read file generator as input.
write back out to file in the subsampled directory.
compress new file(maybe handled by skbio?)
write entry in a subsampled manifest.
Work on scripts to automate the production of test data for the smoke-screen
tests in q2-dada2
.
The first of these are used to subset data from larger fastq files, so that
these smaller datasets can be run in a reasonable amount of time. Eventually
this might be added to dev tools
.
The other essentially hack into the _denoise_helper
function and write the
raw data to .fasta
and .tsv
files when running denoise. This data than can
be used as the expected data for the tests.
:set list
shows whitespaces!
Alright, I got a bit ahead of myself importing the data yesterday. Because I am using the data to run tests, I want to create a subset containing only about 20 datapoints. To do this I want to get random data points from the set:
import random
x = 20
selected_elements = []
while x > 0:
selected_elements.append(
random.randrange((1, 309812, 4))
x -= 1
[140445, 279801, 99949, 128401, 22385, 113897, 7493, 143441, 245621, 220889,
155237, 205217, 92565, 77941, 173313, 181325, 263481, 49605, 178705, 153261]
Then, save these sequences(all 4 lines!) into a new test data fastq file, put this in the manifest and import and run the tests on this file.
qiime tools import \
--type 'SampleData[SequencesWithQuality]' \
--input-path zymo_manifest.tsv \
--output-path zymo_test_data.qza \
--input-format SingleEndFastqManifestPhred33V2
The big push for today is to get @sixvable’s PacBio CCS read pull request for Q2-DADA2 ready. Steps to go include: finishing the paper for the addition of PacBio CCS read denoising to DADA2, making a decision of refactoring to improve DRYness vs leaving as is to avoid monkeying with existing code/improving maintainability (I am leaning towards a bit of refactoring that produces the best of both), creating a subset of the Zymo dataset(as recommended by Ben Callahan, the author of DADA2 and PacBio CCS functionality + the accompanying paper), running the dataset through R and QIIME2 to ensure the same output, and writing some smoke tests that check that QIIME2 is getting the same results as R does (DADA2 is already tested extensively in R).
This is a really accurate method, ~ 50% of 1.5 kilo-base 16S rRNA sequencing reads were completely error free! So accurate that the author encountered too little resolution in the reference datasets. You do have to be careful of chemistry RSII, pre-P6-C4, and SMRT Portal generated reads do not have the same accuracy. Also, almost no homopolymeric repeats in 16S.
The authors were able to detect differences to the strain level, as opposed to the genus or higher level, consistently.
Oxford Nanopore generates longer but less accurate reads.
Short-read: Cheapest, per-sample depth allows for detection of rare community members.
Shotgun Metagenomics: Species and some variant resolution(reliant on suitable reference datasets), can provide information about functional genetic potential. With deep shotgun sequencing: some de novo assembly of community genomes?
Full Length: Combines the targeting of amplicon sequencing with the resolution achieved by shotgun approaches.
Could also be applied in other domains, such as generating complete oncogene sequences, instead of just detecting partial ones, identification of unknown patheogens, and possibly entire ~5kb 16S-ITS-23S region, which could encompass entire viral genomes, as well as a variety of other uses.
Fornutely Benjamin Callahan produced this awesome Github repo of the analysis for the paper!
Retrive mock Zymo sequencing run from NCBI Accension PRJNA521754. The direct download for the raw reads from the sequencer is here.
copy data strip the .1
suffix.
generate manifest file
import:
qiime tools import \
--type 'SampleData[SequencesWithQuality]' \
--input-path zymo_manifest.tsv \
--output-path zymo_raw.qza \
--input-format SingleEndFastqManifestPhred33V2
Working on pre-release activities w/Liz and Evan.
As a side note, if any tag-dev
on busywork is red, there is a good chance
that it is because it has already been run and it does not want to do it
again.
Remember there are pre-built docs on ghost
.
Getting a local copy of a PR:
git fetch $REPO_THE_PR_IS_ON pull/$PR_NUMBER/head:$DESIRED_BRANCH_NAME
git checkout $DESIRED_BRANCH_NAME
I started learning about programming seriously1 as an adult 3 years ago. Not quite to the day, but pretty close. I started off with Zed Shaw’s Learn Python The Hard Way. At the time, this seemed like a great way to get exposed to what programming was all about in as short of time as possible. I still think I can stand behind this reasoning.
I found myself diving into learning about programming for 2 reasons:
I was in the middle of taking classes for a GIS certification and all 3 of the classes for the upcoming semester were going to require progamming, which I essentially knew nothing about.
I try to keep a loose list in my head of things that I “know” and things that I don’t. I like to keep one thing from the “don’t” category moving towards the “know category most of the time. 2018 was the year were the top 2 things on the list were “computers” and “vehicles”.
I have come a long way since then. However, nearly everything that I have done since then has “looked” more or less the same, meaning that I write something up to be called directly from the command line, like nearly all programming tutorials show. I hever have really gotten into the web side of things(though that needs to change soon…). That is to say, I have never made anything that most average people would recognize as a progam or app.
I got inspired to change this during a recent Rust class that I did with some of my coworkers. The class mostly consisted of working through The Rust Programming Language together. did a very simple final project that would give us the chance to use actually build something rather than working problems from the book and rustlings exercises. I choose to do a Pomodoro timer, as this is a tool that I find very helpful for my day to day work.
Coming into this process, I really have no clue about how GUIs actually work. It seemed like you would need some kind of loop to monitor changes to state and some how send these to the screen. Based on some cursory research, this turns out to be the case, but the whole process is pretty complicated…so lets get learning.
Maybe lets not take the use of the word seriously here too seriously… ↩
More carefully worked through SICP exercise 1.19 more carefully and think that I finally at-least sort of grok what is going on. I at least was able to get the correct values for $q_1$ and $p_1$ so that the function works correctly.
Yep, more playing with snippets and $\LaTeX$. I also plan on documenting how I got this setup.
Finished the Real Python Course on modules and packages. I had signed up for a subscription to this site. After trying a couple of other courses on here I cancled my subscription, as the quality of the courses seem to be somewhat variable(with the majority being quite good, but definitely a few that at-least lack in editing and production value) and all of the material feels quite superficial. To be fair, they have really good overviews of topics, but also $20/month will buy you a lot of good programming books that will be much more indepth and that can serve as references rather than just an introduction to the topic. I hope that me feeling this way is an indicator of my growth as a programmer rather than an criticism of the material itself. It is nice to feel like I finally know enough to be able to distinguish whether a particular piece of material is too easy/something I already understand well, just right for expanding my understanding, or will require some pre-requiste study before I can really understand it. I will take it as a mild indicator of competence.
More work on getting snippets, $\LaTeX$, markdown preview, and Conjure/Racket working in my Nvim setup. These are another set of suprisingly time consuming things to get working correctly.
Finished ConjureSchool
and looked over the documentation for
Conjure
. One of the more helpful commands that I found was
<localleader>cs
which starts a new racket REPL when inside of a
Racket file inside of VIM.
Configured NeoVim to work with my material for SICP. I have installed
wlangstroth/vim-racket
and Olical/conjure
. The vim-racket
plugin
allows Neovim recognize .rkt
files, which in turn allows them to be
run using Conjure
. After doing this I also went through the built-in
:ConjureSchool
and also worked on making a cheatsheet of all of the
commands that are used by Conjure
. Switching to a more integrated nvim
experience has left we with lots to learn, but while it initially seemed
very overwhelming, a lot of it already seems fast and very natural. The
learning curve initially seems much steeper than VS Code, but it really
lets me keep all of the things that I liked about vim, but with a much
more polished environment.
Worked on problems 1.16 and 1.17 from SICP. The conversion from a recursive process to an interative one is supprisingly challenging at times. I think that there probably exists a more general, abstract procedure avaliable that would describe that problem encompassed by these two problems.
Python Modules and Packages: An Introduction from Real Python through part 8: Python Packages.
Worked on getting snippets setup and working with
‘neovim/completetion’. At this point I am running UltiSnips
, though
whether on not that continues to be the case depends on how well I can
get it setup with built-in LSP.
Finish :ConjureSchool
Finish the module course from Real Python
More SICP
Exercises from Discrete Math.
Finish setting up snippets, including ones for commonly referenced books.
Exercises from *Introduction to Bioinformatics and Functional Genomics”.
In Simply Scheme I finished the chapter 21 exercises and read chapter 22. Chapter 21 really seemed like a bit of drag/it felt a bit irrelevant/busy work. However on reading chapter 22, it seems like it has at least set some groundwork for other stuff, even if it is only directly applicable in Scheme.
25 minutes of Python Katas on Codewars
Read chapter 21, The Functions
Program from Simply
Scheme.
The section from Introduction to Applied Bioinformatics on the Smith-Waterman algorithm.
The last month or so has been extremely busy, though rather than being with usual holiday season stuff, I have been working a new job out a Sipaulovi Village on the Hopi reservation. This is just a short time gig installing heat pumps for a friends company with federal grant money from the CARES act. It used to be that the residents of the reservations in Northern Arizona could get free coal from the Kayenta mine, however this ended recently with the decommissioning of the Navajo Generating Station in Page. Coal, wood, and propane must all be trucked in, adding to their already somewhat high cost, and it uses vastly more power to get the same amount of heat out of space heater than from a heat pump. Arizona is making a strong move towards solar power, so much of the electricity used to run these will hopefully be much lower impact in the near future. Finally, it can get hot in this area in the summer and these can also serve as higher capacity, quieter, and more efficient air conditioners than most people are already using. On my side, it has been a great opportunity to get to work in a place that very few non-native people get to spend a lot of time. The scenery is beautiful; the crew that I am working with has remarked that the sky really does produce the brilliant, “narrow” sunsets that you see in much authentic native painting. We have gotten to install units in some of the clan house where members of the tribe shared some of their stories, showed us their ceremonial garb, and invited us to view some of their dances once the pandemic has died down.
Finished Chapter 3 of Bioinformatics and Functional Genomics. This chapter focused on pairwise sequence alignment, as a future project I may work to implement some of the described algorithms in Scheme.
Did part of the Pairwise Sequence Alignment chapter from IAB. This went well with the contents of the chapter from Bioinformatics and Functional Genomics.
Read through a Real Python article on Python Modules. :w
Started reading through Introduction to Applied Bioinformatics. While working up at Second Mesa this week, I installed Anaconda and the required packages on a Linux VM. One thing that I plan to do differently than the book is to use Ipython and python files to work through it and then export it to an ipynb, as I like to be able to edit with VIM rather than in a notebook.
Yesterday and today have been filled with lots and lots of work on getting my website looking not completely terrible. Or even just functioning. Somehow I had not realized that the categories
tag was for creating directory structures, while tags
was what I was actually trying to use. This created problems when I tried to included the tag Code: The Secret Language of Computers, as it tried to create a directory with this name, which it could not, and would allow the website to generate.
I have also been working on the organization and styling of the website, which were in dire need of help. I probably should have used a theme, but as with most things in my life, I wanted the experience and control of doing it myself. So I am learning about SASS, CSS, and HTML. Which has been an interesting learning experience but doesn’t exactly lead me to believe strongly that I would like to be a web developer.
Got the play-ttt
function. I am not sure exactly how, I downloaded a
fresh copy of ttt.scm from the book website and deleted and re-entered
the apparent offenders in the function and it seems to be working
fine. I think maybe I had copy and pasted part of the function from
the pdf of the text book and that the apostrophes had the wrong
encoding…but I am not sure.
Read the section on PAM Scoring Matrices from Bioinformatics and Functional Genomics.
Worked problems on Code Wars for 35 minutes.
Chapter 6.1 from “Python for Data Analysis”
Finished chapter 4 of Python for Data Analysis
Read/worked through chapter 20 of Simply Scheme up to start the section
Tic-Tac-Toe Revisited, where I ran into an error where I believe my
square procedure kept returning #f
instead of the function name when the
stupid-ttt
was called.
Bioinformatics and Functional Genomics chapter 2 exercises.
Started Simply Scheme 19.11, the author says it is hard. I feel that. I seemingly have made no progress. I feel as if I can’t quite articulate what I am unable to do. Which I suspect is exactly where the problem lies, that is if I could articulate it then I could solve it. Or rather the difficult part of the problem is either actually saying what I am trying to do, or rather the problem is actually know what I want to do. Isn’t it always. Or maybe I just feel that way because I am reading Douglas Adam’s The Hitchhikers Guide to the Galaxy right now…
So far, everytime I go to write my repeated
function, I end up not being
able to recurse, it is amazing how many times my brain can dream up the
same idea, and excitedly think that it has found a new way of tackling the
problem. Once I do manage to solve the problem, I plan to do a write up of
solving it, as I think that might help others who make it as far as I have
in the book and help cement the ideas in my head.
19.9 and 19.10 from Simply Scheme. I definitely need more practice recursing through tree data structures. I am able to almost get them working but I often feel as if understanding what is really going on is just barely there and I have trouble working through exactly what I need to change to get them working correctly.
Chapter 2 from Bioinformatics and Functional Genomics.
5-2 from Digital Computer Electronics
Chapter 4: NumPy Basics: Arrays and Vectorized Computation of Python for Data Analysis
Worked on integrating LaTeX into Anki for making flash cards with mathematical notation.
- Spent a lot more time on Course Project for [*Genomic Data Science with Galaxy*](https://www.coursera.org/learn/galaxy-project/home/welcome). I am not so sure that this course is worth the time that I am putting into it. While Galaxy seems like a good tool for doing genomic analyses and it does provide good examples of reproducible workflows, I really only was doing the course as a part of the certification. I may end up finishing it when I have actually completed the other course in the specialization.
- Reading: - Chapter 12: Binary Adding Machine from *Code: The Hidden Language of Computers*
Worked on the Course Project for Genomic Data Science with Galaxy.
Finished 18.5! I ended up re-writing my lateral function to do the passing over instead of my depth-wise function, which allowed me to eliminate the blank lists that I had kept ending up with before.
Module 4 and 5 of Genomic Data Science with Galaxy course.
Spent lots of time working on 18.5 of Simply Scheme. I think that the way that I am trying to pass over the parts of the list that I don’t want to keep is the problem.
Completed the Johns Hopkins University: Introduction to Genomic Technologies on Coursera.
Worked on cleaning up the navigation on my website.
The last couple of weeks have been hectic, both nationally and personally. I probably have let the national get too much in the way of personal. I have been getting more studying done, but have been prioritizing it over writing up what I am doing. This is not the most positive, as I do tend to stay on track better when documenting what I am working on.
In addition to the things that I had been doing before, I also have been going over my resume and have started applying for jobs. This has become a bit more of a realistic possibility as Gio has started attending a “SAFE” learning center offered by the school district.
I have been focusing more on the coding side of things than studying for the CCNA as that is the direction that I would like to head, though I certainly am continuing to study for it.
This week has been quite hectic, with a lot of it taken up continuing to winterize our trailer, helping random friends with things, and school for my son. The good news is that my son has gotten into one of the school districts SAFE learning centers and so I should have more time to focus on studying and applying for more jobs. I also have been working on my resume, signing up for job fairs, and other job hunt related things.
Finished through exercise 17.6 in Simply Scheme. So far, this
chapter has somehow felt more “real”, yet not to hard.
17.6 however asked you to implement a function that produced the
same result as the standard append
scheme primitive, only using
the list primitives cons
, car
, and cdr
.
The whole thing seemed pretty straight forward, until I spent over 2
days on this working trying to get it so that not only would it
return a list of the same elements as the regular append
function
but so that it would do it in the correct order.
I still don’t think my solution is as elegant or simplified as it could be, but it does function correctly. And while it doesn’t probably matter that it is optimized, as there is already a function for doing this, I think optimizing it would be helpful for my further understanding of using lists and help me think more deeply about the problem. Plus being satisfying…
Read through pg 46 of Python for Data Analysis.
Exercises through 17.4 of Simply Scheme.
Spent a good chunk of the morning looking for jobs and working on my resume.
The afternoon was spent getting our trailer ready for winter, with the first night below 20°f. We are trying a skirt made from a non-woven house wrap this year. I also covered some of our windows with clear film. The taping for these was tedious, but putting the film on and shrinking it is very satisfying. The windows covered with this film feel much warmer than the other ones.
Read Part V introduction and started on Chapter 17 of Simply Scheme. I was really excited about starting this part because it actually feels like the book is diving into programming and not simply providing a gentler introduction to it. Which is no criticism of the gentle introduction, I feel very well prepared going into this section, but this feels much more substantial and applicative to the “real world” than previous sections have felt.
Continued making the review Anki flashcards for chapter 2 of Wendell Odom’s CCNA guide.
Finished the remaining exercises in Chapter 16 of Simply Scheme.
Started on making the review Anki flashcards for chapter 2 fo Wendell Odom’s CCNA guide.
The big thing was finishing exercise 16.20. This exercise felt quite challenging, as it required understanding exactly how a function that someone else wrote, and then expanding it using a function that was similar required different inputs and a different set of recursions. I may work on a write-up or video of it.
Read Chapter 2 of Wendell Odom’s CCNA guide. While the video courses feel a bit easier and are maybe less “dry”(a common critique of Odom’s book), for me anyways they are much harder to retain information or study from. This book in particular is divided into chunks that are a great size for studying. This makes sense as they are written with studying in mind and have a section at the beginning that tells you exactly how they are meant to be used.
The reading sections of each chapter consist of around 1 - 1.5 hours worth of material, and they have very clear review sections at the end of each chapter. There are also other “review” subdivisions that I will discuss when I get to them.
My plan up to this point has been to read the chapter on one day and then create Anki flash cards on the next day using the review section as a guide. I will detail this process more later. As an aid to the creation of flash cards, I also purchased the Premium Digital Edition of the book directly through Pearson. The print version of the book includes an excellent coupon for this that makes it very affordable.
Finished re-reading the basic Jekyll Step-by-Step Tutorial.
Chapter 1 review as Anki flashcards for Wendell Odom’s CCNA guide
Simply Scheme 16.12 - starting 16.20
Finished Section 21: TCP UDP, video 109. David Bombal’s CCNA Course
The reading portion of Chapter 1 in Wendell Odom’s CCNA guide. I started this because I think I do much better studying from a book rather than just the videos. I plan to watch the videos first to get a high-level overview, then read the equivalent sections of the book. I was driving my self crazy scrolling back to get particular commands or to hear a detailed explanation again.
Section 21: TCP UDP, videos 103-108. David Bombal’s CCNA Course.
Finished Section 20 from David Bombal’s CCNA Course. I forgot about making sure to turn telnet on, for the first video and was pulling my hair out trying to get it to work.
For future me and everyone else who has forgotten how to do this:
line vty 0 4
password YOUR_PASSWORD
login
16.12 from Simply Scheme.
Started working on a post about organizing a website using Jekyll.
Listening to today: VØSNE: Hypnotic Session
Started Section 20: What is a Loopback? from David Bombal’s CCNA Course.
Exercises 16.8-16.11 from Simply Scheme.
Section 19: Duplex and Speed Mismatch Demo from David Bombal’s CCNA Course.
Simply Scheme, pp267-275, including copying out and carefully working through the chapter 16 program in my mind. I feel like I am finally getting my head around this chapter. Bring on the exercises tomorrow!
Went over how to format websites using Jekyll, first practicing on my local “sandbox” website. Actually, very first, I installed Ruby and Jekyll on my Windows machine. I am still really liking the VSCode/VIM combo. I have also started on an article about organizing your Jekyll pages. All of the information is in the documentation, but I think at least for myself it will be handy for the future.
^
jumps or tells the action to
jump to the first character in the line.
Started Section 19: Duplex and Speed Mismatch Demo from David Bombal’s CCNA Course. For some reason I could not concentrate on this at all. I went through the first 2 videos of this section but I think I will be going over them again tomorrow.
pgs 261-267 (up to the Abstract Data Types section) of Simply Scheme.
Finished Section 18: Broadcast and Collision Domains from David Bombal’s CCNA Course.
~18:00-~31:00 of Powershell Master Class: Connecting Commands Together
pg 256 - pg 261 of [Simply Scheme&]
Section 17: The Life of a Packet and started Section 18: Broadcaastnd Collision Domains from David Bombal’s CCNA Course.
A bit more of chapter 16 from Simply Scheme.
Not much today:
Finished through video 81 of Section 15: Cabling and Packet Flows from David Bombal’s CCNA Course.
Worked through part of the implementation Chapter 16 of Simply Scheme.
Read half of Chapter 4 from Automotive Technology: A Systems Approach
There was a bit of a break there as I was preparing for/on vacation with the family. I spent three days before leaving working on our van/ putting in a new bed in the back of it.
We spent the time camping at San Mateo Campground at San Onofre State Beach. For day trips we snorkeled and paddleboarded at La Jolla and then spent several days just playing on the beach, fish, boogie boarding, and surfing at San Clemente and Doheny.
Section 14: Subnetting Vlogs and Section 15: Cabling and Packet Flows How does data flow?, Ethernet History, 10base2, MAC addresses, and MAC Addresses CSMA/CD and Collisions from David Bombal’s CCNA Course
Worked problems from Chapter 16 of Simply Scheme up to 16.5.
Read Chapter 3 from Automotive Technology: A Systems Approach
Today was not particulary productive in regards to studying. Most of the day was spent making preparations for our camping trip to San Onofre State Beach.
*paragraph one*
<p></p>
*paragraph two*
There may be a more elegant way to do it, but this works!
I have decided to really focus on tackling studying for the CCNA for now, with the goal of sitting for it near the end of the year. I also plan on keeping up on my coding. That being said, there will probably be a bit less variety on my progress for a while. I plan to continue working throught Simply Scheme as well as random Python projects and some work on my website. That said:
Finished Section 7 and then Intitial Device Configuration (Section 8) and Basic Cisco Configuration Lab (Section 9) in the Complete Networking Fundamentals-CCNA course.
Installed ‘ipython’ in my base install of python so that I can use it in
the terminal inside of vscode. You just open your Powershell Terminal
inside of VScode and enter python -m IPython
to run it.
Most of the morning I worked more on using requests and Beautiful Soup to find all of the PDFs linked on a page and merge them into a single one on the local drive. It does feel good to be doing this and only having to look up documentation rather than puzzling over how to actually do something. That may be a small step but it does feel like all of the time studying up to this point that I have spent has begun to pay off. I still need to work out how to get it to read pdfs from url’s.
Devel box. I ended up doing this with metasploit and upon looking at the system info I think the reason that I was having so much difficulty getting this to work with any of the payloads that I tried is that it is a 32-bit system and the payloads were 64-bit payloads. Tomorrow I am going to try to find a 32-bit payload and give it another try without metasploit.
The first two videos from section 6 hexadecimal from David Bombal’s CCNA course.
In an effort imolate myself up on the Altar of Diving Down Rabbit Holes, I continued watched Powershell Fundamentals from the Powershell Master Class by John Savill.
Most of the day was spent on school work with the kiddo. A difficult but productive day. The lesson’s from The Explosive Child served me well.
Chapter 8 of The Twilight of the Gods: War in the Western Pacific, 1944-1945.
Worked on creating a custom color palette for my website. Found a palette that I liked and then I saved it in a format that I think will work. That is a late morning project for tomorrow.
Completed the Basic CSS. So much to learn still.
Helped disassemble the cooling system of the racecar. There was nothing obvious wrong with it upon disassembly, though it did not seem to be flowing coolant when we tested it the other day. It is getting a new water pump, new thermostat, and new thermostat housing when Joey gets back. We will probably remove the thermostat from the housing to flush it. As is ever the case, we discovered that to replace the thermostat housing, removing the fuel rail would be necessary. Which may or may not requiring removing all of the headers. We shall hope not.
Today has solely consisted of playing around/struggling with the hack-the-box devel machine. While there are write ups of how to do this machine(including ones that I have done in the past, which I only have a very foggy recollection of), I am trying to work through this as nearly on my own as I can. I think I have a basic plan of attack and have been able to get an unelevated shell by uploading a .asp webshell that I was able to use to run a netcat shell that I hosted on a SMB server. Right now I am reviewing my powershell knowledge(mostly as an excuse to do this, I know that I could just look up the specifc way to do this) so that I can try to run the sherlock.ps1 exploit discoverer from the smb server. I know that I could use metasploit or the windows-vuln python script, but I would like to see if I can get it to work doing it this way.
Reading:
There are a lot of strategies in here that I think will be helpful for our somewhat aspie son. It is a bit funny as what the author (or at least his editors) describes as an “explosive child” seems to fit the description of children with Aspergers (mild ASD according to DSM 5) exactly. It does make me wonder if many of the children labeled as “explosive” or “difficult” really are Aspies at heart. A book title Learning to Mitigate the Meltdowns of Your Child With Asperger’s would sell considerably fewer copies though.
Either way, it appears that many of the strategies and the mindset that the author asserts are sound. He says they are backed up by years of academic research. I haven’t had the chance to read the research he is refering to. Or even to see if he has included citations in the back of the book…yeah, I need to do that.
However, trying to adopt the mind set that children that act like this do not possess the tools to deal with many situations and therefore the goal to have as a parent is to try to help the child gain the tools to deal with a situation appears to be sound. I tried to engage this strategy as best I understood it at bed time and to a situation involving a broken flashlight. It seemed to be quite effective, he did a much better job dealing with both situations when I was able to help him figure out how do the things that he either wanted or knew that he needed to do. The topic of problem solving may require its own post or series of posts in the future.
Overall I am quite impressed so far and highly recommend it to any parents with either Aspies or “explosive” kids.
- Chapter 8 of [The Twilight of the God](https://amzn.to/3hZRUmR).
layout: post date: 2020-09-21 tags: [scheme, programming, recursion]
Got the straight?
predicate for the scoring poker hands project
working.
Section 5: Binary from David Bombal’s. A lot of this was review, but I wanted to make sure that I didn’t have any gaps in my knowledge for later in the course.
Spent 2 hours playing with layouts and organizing pages on a “sandbox” website to learn and practice organizing and formating pages with Jekyll.
Later, I would like to investigate whether it is possible to save links on Jekyll sites so that I don’t have to enter the actual link over and over. An example of where this would come in hand would be the links to the various udemy courses that I am doing that I post each day discussing what I did from the course.
Additionally, I would like to see if there already exists the functionality to automatically update a reading list from things I say about the book or article in a post.I doubt that this already exists, but would be a good project or challenge to have for myself. This would also be handy for things like automatically updating my Markdown Tips page from inside of posts.
Started trying to work through Devel from Hack the Box without looking at any guides. It definitely is a lot more challenging to work through them from scratch, but a lot more fun! I have tried several things out so far and I at least think I have an idea of where I have gone wrong. I plan to work on it more first thing in the morning.
cracking the hash with Hashcat from Practical Ethical Hacking. This ended up being more complicated than I thought as I ended up needing to install NVIDIA CUDA and Visual Studio to try to get it running right. I was not able to get it running successfully, I think my graphics card may be too old and not compatible with hashcat. I am working on building a new computer and hopefully this will not be an issue soon.
Left for a backpacking trip up Humphreys Peak in the Kachina Peaks in Northern Arizona. We ended up getting started hiking around 1600, which is definitely late, however, we arrived at tree line where we made camp less than half an hour after dark.
Reading:
Chapter 7 of Twilight of the Gods (affiliate link). This chapter discusses various components of the cumulative war against the Japanese during WWII. This is in oposition to the typical incremental narative that is more concrete and more commonly known. The chapter discusses the birthpains of the USAAF’s 20th Air Force’s strategic bombing campaign against the Japanese home islands, a broad strokes overview of the allied submarine commerce raiding campaign against the Japanese merchant marine, filled out with the stories of a few noteworthy examples of this type of warfare, a brief discussion of the saga of the US Navy’s torpedos, and the narative of the sinking of the the Shinano (the largest warship to ever be sunk by a submarine).
2.5 hrs getting packet tracer setup and exploring the TCP/IP model basics, then taking TCP/IP model quiz from David Bombal’s CCNA course.
Started a music page for my website of some of my favorite music.
Spent a long time working towards the Scoring Poker Hands project from Simply Scheme. I started defining the master function and also a predicate function for determining if it is a straight or not. I may just be really tired, but this has been quite a bit more challenging than I thought it would be.
Life is more complicated: Tips and Tricks, All four videos(17-20) for the Practical TCP/IP Model, and the TCP/IP model quiz from David Bombal’s CCNA course.
Section 17: Introduction, LLMNR Poisoning Overview, and Capturing NTLMv2 Hashes with Responder Heath Adam’s Practical Ethical Hacking Course.
Networking Devices: Firewalls, IDS, IPS, and WLC and OSI and TCP/IP Models from David Bombal’s CCNA course
Installed VM Windows server and client machines. Setup an Active Directory Forest. This was very challenging for me at this point, as I have not ever actually worked with Active Directory before. It is something that I have known is important and I am very excited to be getting to work with it. The process that I actually went through is detailed in Setting Up the Domain Controller, Setting Up the User Machines, Setting Up Users, Groups, and Policies, and Joining Our Machines to the Domain from Section 16 of Heath Adam’s Practical Ethical Hacking Course. Except that I used a Lord of the Rings theme. I am really excited to start in on the exploits tomorrow.
Replaced the tempered glass screen on my tablet after stress cracking it the other day. I nearly completely messed it up with several pieces of dust falling onto the screen between removing dust with tape and laying the screen down, despite being all of the about 8 seconds. The dry air here in Flagstaff can make this challenging. It took around 4 tries to get it all off, which is risky, as you risk removing the sticky from the protector or cracking it, but can be done very carefully.
Read the chapter on formatting in Jon Duckett’s HTML and CSS (affiliate) in preparation for working on a body template for my website tomorrow.
Why Structure and Interpretation of Computer Programs Matters
Read more about how to apply styles across Jekyll sites. I created a “sandbox” site to play around with and begin developing my own theme. While this is not my intended career direction, it has been really fun to do. I also really feel like I will be able to get my site to look the way I want, which was never the way I felt about any of the other
Continued working on Simply Scheme Project: Scoring Poker Hands. I have found this exercise very challenging. From a top down perspective, I have either figured out how to or actually written the code to implement the program. I also have been able to write the code to determine flushes and the various pair hands.
However, I am strugg working through a function to determine
straights and to sort the cards by rank. More specifically, I think
I understand how to implement a sort, but only if I have boolean primitive
that I can use to determine which of two values comes first. I am sure
that there is an easy solution that I am missing. Normally when I feel
like this, I have missed something simple. To that end, I am re-reading
the entire recursion section (section IV). I some ideas on how to
implement this boolean function, but hope clarify my thinking on this into
something useful while reviewing.
Markdown notation of the day the strike through, by ~~strike
through~~
.
Second Markdown notation tip of the day: You can create multi-paragraph list items by including a blank line between the first and second paragraphs of the list item and being sure to include at least one space before the start of the second paragraph so that markdown will indent it as part of the list.
vim
command of the day: Jump to the end of the previous line: k$
.
Not really an independent command, or really difficult, but useful.
Began working on a script that would find and combine all of the PDF
files on a webpage. It may develop into full blown python module at some
point, but at this point it is going to live as a single python
script
in my personal Github repository. As ancillary parts of this/general
coding knowledge, I will need to install and begin learning
BeautifulSoup
and Requests
. At least.
Read through a lot of the Jekyll websites documentation. I have both some formating, functional, and stylistic changes that I want to make to my website that I hope to be able to implement after reading the docs and probably learning a bit more about CSS and HTML.
Articles Read:
The topic of escaping restricted shells I found really intriguing from yesterday’s Linux PrivEsc article, and I was able to find this Escaping Restricted Linux Shells article for today.
Worked on the racecar cooling system again. After about 10 years of only running water as the coolant, there is tons of rust and scale built up in it. We rinsed it again and then let some CLR sit in it for about 3 hours. It removed a ton more junk, but there still doesn’t seem to be much flow through the cooling system. We did finally get a temp reading on the coolant temp gauge inside. At this point we are going to ask on the E30 forums if there should be coolant coming out of the bleeder immeadiately or not, which might change our approach. Barring any good suggestions from the query, we will be replacing the thermostat, thermostat housing, and water pump, as well as flushing all other lines directly.
Section 15 Active Directory Overview in Heath Adam’s Practical Ethical Hacking course on Udemy. Also downloaded ISO files for the Active Directory lab build.
Read these articles:
Installed Cisco Packet Tracer on my Windows 10.
Watched through video 13 Networking Devices: Repeaters, Hubs, Switches, and Routers of David Bombal’s CCNA course
Generating Shellcode and Getting Root in Heath Adam’s Practical Ethical Hacking course on Udemy.
1.5 hours of work on the Scoring Poker Hands project in Simply Scheme. Produced functions to deal with hands produced by matching ranks. Next session I plan to develop a function to sort the ranks into appropriate order and then score flush hands. Finally I can produce a master function that will be able to determine the value of any given hand.
Reread Chapter 2 of Simply Scheme for review.
I was being driven crazy by the system beep tone whenever I accidently
entered an invalid keysequence or tried to tab to a file that didn’t
exist. I am running Kali Linux with the XFCE desktop. To turn it off, you
can open a terminal and enter: xset -b
. To permanently turn off the tone
I created a startup program by going to Settings Manager -> Session and
Startup -> Application Autostart -> + Add
With the settings:
Name PC Speaker Off
Command: xset -b
Trigger: on login
vim
command of the day: %
is used to jump to a matching parenthesis,
square bracket, or curly brace, the start or end of a C-style comment,
as well as other matching keywords/conditionals if they are specified in
ftplugin files.
freeCodeCamp’s Responsive Web Design Certification Adjust the Margin of an Element → Improve Compatibility with Browser Fallbacks
Worked through HTB: Lame without metasploit as OSCP practice. I ended up writing my own python script for, though it looks a lot like other available ones, though I wrote it for python 3.
A few further notes. An important part of the script is assignment of the shellcode with the call:
userID = "/=` nohup " + buf + "`"
There are two important things about this. The first does not actually
directly relate to this call itself but rather the msfvenom payload
referenced by buf
. You need to be sure to delete the b
at the
beginning of each line that will tell python that it is byte code. By
removing this, python reads the line as a string and can therefore
concatenate it with the other strings you have supplied. The second is the
call to nohup
which is a linux command for “no hookup”. As best I can
tell, this is because we are trying to run our code between entering the
logon userID(which is what contains our code) and the system asking for
a password. To get the required time, we provide this command, which then
lets our other code run in the background. I will also include a few
resources that I read through after running through the box as much on my
own as I could:
Finding the Right Module in Heath Adam’s Practical Ethical Hacking course on Udemy.
mona.py
script. I had issues with cURL when trying to download it in my
Windows 10 environment. When I come up with a solution to this I will post
it. My second attempt was to simply use vim
to create the file directly
in the directory. However, on my new install of Windows 10, I couldn’t
paste the text into vim. Look below for the fix to this.In the afternoon I helped flush the coolant system on a friend’s new 24 Hours of Lemons racecar. While flushing, we discovered that the coolant system has several leaks and also does not seem to be running coolant completely through the system very well. A “new” waterpump may be in order. In this race series, that cost of the vehicle less the cost of safety systems must be less than $500, which doesn’t leave much room for anything approaching actually new parts.
Open Powershell as an Adminstrator, then use the vim command:
:edit $MYVIMRC
which will edit your .vimrc whereever it is then added this to it:
source $VIMRUNTIME/mswin.vim
behave mswin
and then save via the usual route.
Not much actual learning got done today, but it was fun:
Experimented with different themes for the blog. Some of them wouldn’t work with Github Pages, including my favorite one. That was a little disappointing, but provides a learning oppertunity, as I will probably try to write a similar but compatible one!
Created a Google Analytics account. I did not get it setup on my website, but at least I will be able to soon.
Finished selling one of our cars. Not fun, but exciting.
Enjoyed the second day of very Autumnal weather with a bike ride with the family. We rode to downtown, got coffee and muffins, and then rode to Buffalo Park where we took a nice walk. Bike rides in Fall weather are one of our favorite things!
Had Buffalo Chicken and Instant Pot Mac and Cheese while watching Star Wars: A New Hope with one of our friends.
Transfered domain registrar from Wordpress to Google Domains, set up DNS and an enabled https.
Worked through and took notes on Finding the Offset, Overwriting the EIP, and Finding Bad Characters in Heath Adam’s Practical Ethical Hacking course on Udemy.
Worked through and took notes on Introduction to Buffer Overflows, Spiking, and Fuzzing in Heath Adam’s Practical Ethical Hacking course on Udemy.
~2.5 hours reading through the previously mentioned Building a Static Site with Jekyll and Github and began working on setting up my own site, which mostly consisted of installing Jekyll and generating the basic files.
Freecodecamp’s responsive webdesign course Basic CSS: Introduction to Basic CSS -> Adjust the Padding of an Element
2.5.2 Overview of How DNS Works of Networking: A Top Down Approach (Affiliate Link)
Not much on today, most of the limited time I got to work on anything on the computer was spent trying to figure out how to setup a website with Github pages, as I had essentially quite writing on my wordpress page because their “fancy” layout drove me crazy and I much prefer being able to write using vim and markdown, especially if it is combined with git/github. Apparently sometime in the past I created a repository for this but never set anything up on it. Well this evening I gave it a go, though that essentially consisted of writing a short intro and trying to figure out how to view it after deleting the default index page. Apparently I need to learn anything about how a website is actually setup, so I found the articles below that I plan to read when I get done with this. I want to have a site that is somewhat of a blog/homepage hybrid, hence the article choice. With that being said the day consisted of:
Write introduction, activate github pages, delete default index file.
Wonder how to actually display anything.
Setup my .vimrc file on my new Kali install.