Category Archives: Uncategorized

1 – webvr scrape

Little experiment to scrape web vr sites.

Getting started -> using scrapy on Windows.

Python tool.  Downloading current Python (3.7.1): https://www.python.org/downloads/

Python is accessed via “py” not “python” on command prompt and pip was only available at C:\Users\[Your Name]\AppData\Local\Programs\Python\Python36-32\Scripts.

Set Windows environmental variables to access pip.

Install virtualenv using pip so projects are standalone with all necessary libraries.

pip install virtualenv

Create a new virtual environment.

virtualenv vr_scrape

New vr_scrape directory is now in the location where command was executed.

(Good summary of the roles of pip, virtualenv and the activate script here: https://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/)

Activate the shell so it uses the new virtual environment (vr_scrape)

[path to vr_scrape environment directory]\Scripts\activate.bat

Install scrapy into this environment (currently active) in shell:

pip install scrapy==1.5.1

And then if you just type “scrapy”:

Scrapy 1.5.1 – no active project

Usage:
scrapy <command> [options] [args]

Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy

[ more ] More commands available when run from project directory

Use “scrapy <command> -h” to see more info about a command

To create a new project called “A_initial”:

scrapy startproject A_initial

Scrapy also seemed to need this package installed:

pip install pypiwin32

Sublime  text 3 which required this new build system file :

{
“shell_cmd”: [“C:/Users/[USERNAME]/vr_scrape/Scripts/python.exe”,”$file”],
“selector”:”source.python”,
“file_regex”:”file \”(…*?)\”, line ([0-9]+)”
}

And then this “hello world” spider to scrape BBC news…

import scrapy

class BbcNews(scrapy.Spider):
#identity
name = “bbcnews”

#requestscr
def start_requests(self):
urls = [
“https://www.bbc.co.uk/news/world/africa”,
“https://www.bbc.co.uk/news/world/asia”,
“https://www.bbc.co.uk/news/world/australia”,
“https://www.bbc.co.uk/news/world/europe”,
“https://www.bbc.co.uk/news/world/latin_america”,
“https://www.bbc.co.uk/news/world/middle_east”,
“https://www.bbc.co.uk/news/world/us_and_canada”

]

for url in urls:
yield scrapy.Request(url=url, callback=self.parse)#callback will parse the response

#response
def parse(self, response):
region = response.url.split(“https://www.bbc.co.uk/news/world/”)[1]
#response will be same as request url…
#then split by “https://www.bbc.co.uk/news/”
#we get this: [“”,”africa”] and then index 1 (2nd item in array)
_file = “{}.html”.format(region)
with open(_file, “wb”) as f: #wb write byte… response.body returns bytes
f.write(response.body)

This command (inside the environment folder) to scrape it…

scrapy crawl bbc news

Which then dumped out these files:

Oculus Connect 2 Highlights

Some notes on this developer focused conference (watch the whole thing here if you have the time:  -> 1.5hrs):

  • What Michael Abrash is thinking right now – well worth watching for anybody making content.  I love his visionary talks.  This starts around 1 hour 3 minutes into the overall keynotes video.  “These are the good old days…”.  These are exciting times, but the massive challenges to get things perfect is highlighted (taste, sound, vestibular, vision and haptics).  Summary of recent findings and research approach.

  • Here’s an interesting slide on how far vision would need to go in order to be near perfect.

  • The Oculus Studio’s VR short Henry” is released on Oculus share for you to try at home if you have a DK2!   Also – they’ve released the project so you can see how it was built!  And so you can rebuild it yourself in Unreal Engine 4 – this is really cool since Unreal Engine 4 is free to use unless you are making money from it!  Great study material for anybody building content…

  • On the cheaper/mobile end of the market, Samsung have redesigned Gear VR to fit every 2015 Samsung phone – a vote of confidence then.  This is not the unit that those who want to experience the best will go for, but this is arguably the most important market for many software developers as the number of users who are potential customers for software is much greater…
Samsung Gear VR

New Samsung Gear VR

  • Samsung’s device is going to be $99 – that’s cheap compared to prices we have seen so far.  Don’t forget that this device has more features than if you just stuck your phone in a holder with lenses and a strap…

  • Will be ready for 2015 Black Friday… (November)…

  • Mobile software development kit (SDK) for mobile is going to version 1.0

  • Oculus Arcade: Sega, Warner Brothers Midway and Bandai Namco.  This was shown.

  • The increasing availability of traditional media (rent films etc).  E.g. Netflix made available now… Hulu etc coming soon – the big boys.

  • Tighter integration with Facebook and Twitch video sharing/viewing

  • To run the Rift (the flagship unit) will require powerful PCs.  Oculus is pushing the standardization of hardware specifications to make it easier for consumers to buy, sellers to sell/market and developers to predict likely behaviour of their software.  Recommended spec: NVIDIA-GTX 970/AMD 290 or Greater, Intel i5-4590 + and 8GB+ RAM.  “Oculus Ready” badging.  Initial partners Dell/Alienware/Asus.  Targetting < $1000 price point.

  • Rift SDK 1.0 release date – if developers build with this then their games with be ready for Rift release day…

  • Minecraft Windows 10 will support Oculus – quite a turnaround.  It’s past history now, but it was common knowledge that Notch (original creator of Minecraft – now sold to Microsoft) was not a fan of Facebook.  Noted that we have also seen many Hololens demos.

  • Touch controllers will include a 2nd sensor to increase the area the system will be able to track.