[J-core] Jcore mailing list and turtle board & Space processors

[David Summers]

Rob,

Its odd, although I work in the european space industry, didn't intend 
for discussion to go in this direction. E.g. although space processors 
is one business model, it doesn't to me sound like j-core business model.

On 08/07/17 22:17, Rob Landley wrote:
> On 07/06/2017 03:12 PM, David Summers wrote:
>> Rob,
>>
>> I'll just reply to the bits that aren't covered in conversation with
>> Jeff ...
>>
>> On 05/07/17 21:58, Rob Landley wrote:
>>>> Alas here at work, we go for the high end stuff; and often anti fuse
>>>> FPGA - such is the space industry...
>>> We talked about adding SECDED to our DRAM controller last year, but
>>> current customers don't need it. And ITAR means all the US space guys
>>> are read-only consumers of open source and thus developers in Canada and
>>> Japan never hear from them. Oh well.
>>>
>>> http://www.thespacereview.com/article/528/1
>>> https://en.wikipedia.org/wiki/International_Traffic_in_Arms_Regulations#Effects_on_the_U.S._space_industry
>>>
>> Well ITAR is your side of the pond, not ours!
> Ah, not working for the _US_ space program. That explains it.
>
> (In the 1990's a bunch of cryptographers voluntarily gave up their US
> citizenship to keep working on cryptography. And they they inflicted
> that nonsense on the space program just as the cryptographers got out
> from under it.)
Well even cryptography didn't escape ITAR, IIRC the RSA algorithm was 
put under ITAR at one stage. People used to go around with T shirts with 
a perl implementation of RSA on the front, and technically the T shirt 
couldn't leave the country.

Other amusing ITAR issues - have you ever tried using a GPS on a plane. 
Well if a GPS obeys COCOM limits, then its apparently not dangerous. 
e.g. faster than 1000knots and over 60,000 feet. Means its a hassle 
getting a GPS for use on a space craft, which is a pity as it would be a 
good method of locating LEO satellites. Of course, there are GPS outside 
the states that don't apply COCOM limits - but it limits the market.
>> My undesratdning though is
>> that ITAR is typicaly attached to devices, and not alogoriths. So if we
>> take a SECDED algorthm such as Hamming[8,4], I think its just the chip
>> that impliments the algrothm that gets ITAR - and not the Hamming[8,4]
>> algorithm.
> It rates software as a munition. A software implementation is a device
> according to that nonsense.
>
>> Whats kind of interesting, is ITAR is one of the motivations behind
>> ESA/Gaisler to do a VHDL of SPARC-V8 ISA. Here in europe we had
>> relativlt few fault toleant CPUs that were avaiable, ERC32, and there
>> was one other whose name slips me now.
> We mentioned in our presentation that they looked at Leon Sparc before
> doing j-core, and used that project's "two process" method of VHDL
> design in doing j-core.
>
> The problem is sparc's ISA requires way too much memory bus bandwidth.
>
>> Now fault tollerant cpu can typically survive one bit flip, and correct
>> it.
> SECDED: Single Error Correction, Dual Error Detection. (Published by a
> Bell labs guy in 1950, totally out from patent and they can't
> retroactively classify it either.)
>
> We were looking at adding it to our dram controller because we were
> redoing the dram controller at the time. (New one's twice as fast.)
> Other bits of the chip would need something similar, but usually they
> just have multiple instances run the same code and compare the results
> and achieve a quorum.
Yes it was Hamming in Bell Labs IIRC, he effectively created the subject 
of error correcting.

Now I know why we need it in space, but whats interesting is why were 
you looking at it. E.g. dram is usually fairly reliable, so why would 
you need to error correct - given the overhead (e.g. hamming(8,4) has 4 
data bits encoded in 8 ram bits - and is SECDED). Thats overhead we can 
accept in space, but can't see other users would need it - e.g. it would 
need to be very noisy.
>> It means its technology that can be used in ICBMs - so fault
>> tollerant cpus from US typically do have ITAR. and ITAR is far more
>> paper work than is fun going through.
> An ICBM is in space for what, 15 minutes? And the "ballistic" part of it
> points out that at least initially they were basically chucking a big
> rock and calculating where it would come down based on physics. (A nuke
> doesn't have to be _that_ precise.)
>
> A satellite is in space for _years_. Far more interesting problem. Japan
> has a nice space program, pity Elon Musk came in offered to launch stuff
> at a loss (half the price of any other bid; his salesbeing said that
> before the other people had _made_ their bid) to drive out
Can't say we worry *why* a device is under ITAR, only *if* its under 
ITAR - and that becuase of the hassle of paper work that ITAR good require.

Means its rare for say a RAD750, to fly in europe. Yes a nice PowerPC 
chip - but too much hassle and $$$$.

Space X for lauchers is actually interesting, in the past launchers have 
been so expensive, both for development and for flight, that its only 
really been government that have funded (or via ESA in europe).  Space X 
looks like a genuine operation (e.g. they launch things, and not too 
many blow up) and its effectively the first launch operation by a 
private company. It means the market in launchers is starting to open 
up. That has to be a good thing, and thats whatever happens to Space X.
>> So ESA developed the LEON, the SPARC-V8 open CPU design; Gaisler
>> continued the development, to several fault tollerant designs (majority
>> voting flip flops etc). This means we have reasonable poerful flight
>> CPUs avaible here in Europe.
> All for it. As I said, we looked at that before doing j-core. The
> problem wasn't the chip design, it was the inefficiency of the sparc
> instruction set. (Your memory bus and l1 cache are generally bottlenecks
> at the best of times. Being able to fit twice as much code in each cache
> line is a _big_win_.)
>
>> But there it took space flight to develop an open CPU; so it amuses me
>> that at least some of your motivation is CPU for synchrophasors. So two
>> open cpu designs, but with motivation from opposite ends of the
>> applications!
> Moore's Law's brought the price of FPGAs and development workstations
> down. You can squeeze our design into a dinky $50 off the shelf fpga
> board (Numato Mimas v2) and we're making a realy _nice_ FPGA board that
> can handle an SMP instance of our SOC for about twice that (if that darn
> hardware order ever unblocks, grrr). The 8x SMP laptop with 16 gigs ram
> I run my copy of the VHDL tools on was something like $1300.
>
> This was very much _not_ the case 15 years ago. Your entry level systems
> on both sides were multiple thousands of dollars. Not exactly hobbyist
> territory.
Yes. Also much to be said for Xilinx, as they pushed the memory based 
technology. Here most FPGAs were anti fuse, now whilst at $50 for an 
AntiFuse FPGA was far cheaper than an ASIC, it still meant that each 
time you got the code wrong - you had to pull the chip, and flash a new 
chip.

Xilinx with their memory based FPGA, all you have to do is reflash the 
chip - and that really has made development far easier.
>>> ..
>>> Last I tried using OpenOCD to flash something on a regular basis (the
>>> Tin Can Tools nail board), using the fully open source drivers was
>>> _really_slow_. I mean amazingly slow. There was a binary-only module you
>>> could use to speed it up, but even then it was still pretty darn slow.
>>>
>>> *shrug* Maybe it's improved since then...
>> well openocd is 10 years or so old.
> And has not advanced that I've noticed.
>
>> However yes its iterface is
>> horrible. So last time I used it (few years ago) it wasn't nice.
> If even you haven't used it in years, why are you advocating its use here?
Think my question was mainly if JTAG was an option. Now if talking about 
JTAG, you kind of have to talk about what can talk JTAG. And problem is 
that in the open market - its a bit thin. Yes it would be nice if new 
tools had come along since openocd - but I've not heard of them. So 
reason for mentioning openocd is two fold:

  * Its available
  * Its free

Yes maybe there are other options.

>> Speed, well jtag its the prgrammer that sets the speed. need to make
>> sure its not too fast, as the device may not cope. But this means the
>> default speed in openocd is often quiet slow, as then you can check you
>> can talk to the device. Once you can talk though - you can up the speed,
>> at least until things start falling over ...
> The atmel programs the thing pretty snappily.
>
> And the setup is 100% portable: I'm often out with a battery powered
> netbook, turtle board powered by USB cable (I have one of the
> prototypes), and a USB SD card programmer (because the one built into my
> netbook filled up with cat hair over the years). It's a much bigger ask
> to set up a jtag connector in a coffee shop, let alone fit it in my
> messenger bag.
That was exactly why I talked of the FTDI FT232H chip - adding it to a 
board would maybe cost $5. It has a USB 2 interface - and so you just 
plug it straight into a USB port ...

As for programmer - I wonder how hard it would be to write some code for 
talking across the USB to the MPSSE to program the xilinx.

So more compare the Atmel AVR boot chip to the FTDI FT232H chip. I'm 
guessing its easier to use the Atmel to program that flash, and also 
have the flash program the xilinx. FT232H would give jtag, if anyone 
wanted to use it.

So just playing around with ideas.

Guess though if the jtag pins are on the header - a use can access them 
anyway with an FTDI cable, and that would only be $20. So quite possible 
with the turtle board as is.
>>>>> C) Requiring openocd setup as a hard prerequisite would probably
>>>>> eliminate about 2/3 of our userbase. (That's why things like Numato
>>>>> don't. Nor does our in-house EVB because customers wouldn't do it.)
>>>> Ah - must admit I started using it becuase it is open! So anyone can
>>>> download it. Horrible interface though, but then again when doing JTAG -
>>>> one shouldn't expect it to be easy ;)
> Open source things often have horrible user interfaces. It's a
> structural problem with the open source development model. I've
> explained why in talks ala:
>
>    https://www.youtube.com/watch?v=SGmtP5Lg_t0#t=11m30s
>
> That said, some open source UI is worse than others. Things with a large
> userbase of people who also program tend to get enough feedback to at
> least file the rough edges off. Things requiring significant domain
> expertise that _doesn't_ necessarily overlap with software application
> development... The gimp is very much not photoshop. Crypto code is
> eldrich and creaky because normal programmers introduce side channel
> attacks if we touch anything. The git's development has been dominated
> by kernel programmers who should not be allowed anywhere near userspace.
> And with openocd you need to be a hardware dev _and_ a programmer to
> poke at this code.
>
>>> We don't expect jtag to be easy. We do expect using our board to be
>>> easy. That's why we made an easy way to reflash the board.
>> Understood. Yes different user case.
> Not necessarily. It would be great if we could level people up into jtag
> development. We just refuse to have it as a prerequisite.
>
> The simple path we have mapped out for turtle (which requires me to redo
> the website but I _plan_ to when the darn hardware shows up) is:
>
> 1) Download a j-core bitstream binary.
>
> There's a whole sidequest about building it and digging down into the
> VHDL which is its own page and you can go down that rathole at your
> leisure (with "VHDL toolchain install" page, "jcore build system
> walkthrough" page, and "learning to program in VHDL with the 2-process
> method and our coding style" pages.)
>
> 2) Download a vmlinux binary with built-in initramfs. (More sidequest
> pages about installing or building cross compiler toolchain, building
> userspace and kernel from source, explanations of nommu programming and
> ELF vs fdpic and such, maybe some stuff on qemu...)
>
> 3) Boot to Linux shell prompt.
>
> 4) Build your own binary, copy it to the board, and run it.
>
> The jtag stuff would be yet another sidequest. There should be pages on
> it, but it's not in the critical path to start _using_ the thing. It's
> more or less in the "fun with GPIO pins" or "how to use gdbserver" pile.
Actually thats one of the more inspiring things that you at se are doing.

Viz, it isn't just developing the j-core. Its also about running linux 
on the j-core.

Its that you havn't just got it running, but also made changes to GCC to 
do fdpic. Paid for additions to musl so it can be used on cpus without 
memory management.

So you are developing the whole system, and not just the cpu; and trying 
to do it in a way that it will be maintained.
>>>> Suprised though that not used in-house. Here its the first thing we go
>>>> to - means that a chip that is otherwise dead, can be reprogrammed.
>>> We have an atmel boot processor that can reprogram a chip that is
>>> otherwise dead. Built in.
>>>
>>> Keep in mind we have _3_ levels of jtag todo items. There's "jtag talks
>>> to board hardware at xilinx/flash level", "jtag talks to j-core SOC",
>>> and "gdb can single-step our processor using upstream vanilla gdb
>>> source".
>> Yes - and its understanding the different levels that is needed.
> There's a dependency tree here. It's hard enough to explain to people
> the different layers that are already there.
>
> Installing your app means modifying the root filesystem (which can have
> dependencies on what's installed in the root filesystem, but also root
> filesystem format: we're using external cpio archives to load the
> initramfs so they can be swapped out without replacing the kernel, but
> you still can't incrementally modify them EXCEPT in the most recent
> toybox release I changed the cpio generator
> (https://github.com/landley/toybox/commit/32550751997d) to not add the
> TRAILER!!! entry, so you can concatenate the uncompressed cpio archives
> and thus yes you _can_ append to an existing one but that's recent.
>
> In any case it's easier to stick your new code in the vfat partition if
> you want it to persist (modulo naming weirdness). So you need to mount
> the vfat partition of the sdcard somewhere you know about it.
>
> Those two filesystems live on top of the kernel; your kernel config has
> to support them, your kernel command line may need stuff, your init
> scripts need to call the right setup stuff, your kernel command line and
> bootloader setup need to point to the right stuff... we provide an
> example, but it's nice to understand what you're doing here (and that's
> not even talking about _modifying_ the kernel and/or understanding the
> relevant kernel source).
>
> Building this requires a toolchain! With a C library. We're using
> musl-cross-make for that but there should be a walkthrough explaining
> now that works at some point. And of course cross compiling sucks:
>
>    http://landley.net/writing/docs/cross-compiling.html
>
> If you build an app you'll currently need to cross compile it, which is
> fun. We have a todo item to get codelite working, but I only do Linux
> and they want it to work from macos and windows as well, which means
> every time I sit down to poke at it there are these two giant unknowns
> and what seems trivial goes back on the todo list because of that. Any
> time there's a "I could do this simple thing..." "Oh but that's useless
> because I want much bigger thing". "The perfect is the enemy of the
> good, the good has been killed, moving on." And thus I have yet to learn
> nontrivial uses of codelite. It's on the todo list...
>
> Oh, and there's how to program for nommu systems so you _can_ write your
> app (http://nommu.org has a bit of that, I need to push that into the
> Linux Documentation directory, and write an fdpic document, and write up
> warnings about how Rich Felker insists on breaking nommu on musl-libc
> and how to patch his source to work around that when building your
> toolchain...)
>
> All of this needs an "installing Linux stuff onto the board" layer;
> network mount, ftp server, wget, popping out the sdcard and copying
> files onto it, etc. (Popping out the sdcard gives you an OS-level
> "unbrick the board" option if you try to boot an unusable vmlinux.)
>
> Now we dig down to the bitstream layer. Let's skip the "building it"
> part and the "developing it" part, and just talk about the "installing
> it" part. On power on, the atmel chip loads the FPGA from the spi flash,
> so to update you write the bitstream in the spi flash and reboot. In
> theory we could reflash from within Linux (older systems forced that
> read-only by holding a line high, but the current one changed that. I
> don't remember if it's in the 50 mhz release, but it should be in the
> 62.5 mhz one), but if that gets interrupted it can't load a good SOC
> image to run vmlinux to get to the point where you try again. I.E.
> brickable.
>
> We already need the atmel chip to load the fpga image from the SPI flash
> at boot time, so we also taught it to _write_ the SPI flash. It already
> needs to be there and be able to talk to the spi flash, this is a minor
> repurposing of existing hardware.
>
> In theory, you can also do this flash writing with a jtag, but nothing
> up to this point has required the jtag to exist, and you still need the
> atmel to _read_ the spi flash. We added the
>
> You can also use a jtag for debugging, but there's like 5 other ways to
> do that. We've occasionally pulled out one of the inner logic states and
> run it to a GPIO pin or an LED, which is the hardware version of
> sticking a printf into the code. We also have a simulator version (ghdl
> or nvc) where we CAN stick a printf into the code.
>
> So getting back to the "many layers" thing, when you fire up the jtag,
> what guarantees the FPGA has been programmed? Talking to the spi flash
> when the FPGA isn't running yet, vs talking to the j-core SOC when it's
> enabled but not running anything yet (kernel or bootloader), vs using
> the gdb protocol (ala qemu -s) to "debug" the processor, vs using that
> same gdb protocol to talk to the Linux kernel (kgdb) or a userspace
> process (gdbserver) without needing a jtag...
>
> If you're wondering why this is nowhere near the top of our todo list,
> it's because there's literally dozens of individual things higher up the
> list just in terms of writing documentation for j-core.org to explain to
> people how to do this thing, and where it fits in Giant Dependency Tree.
>
>> That is both the stength of jtag, but also what makes it a nightmare to
>> use.
> If you look at the different layers in context, the jtag isn't really
> necessary. If you've got an FPGA and/or software simulation of your VHDL
> soc, you don't need the jtag to view/control the internal state. (Heck,
> we can run a control line out to the GPIO pins to stop/start the
> processor clock, and if we use two pins we can trivially single step it.
> We just haven't bothered yet.)
I suspect Jeff is just very good with the VHDL. Bit like with ARM, when 
the first ARM1 was made, it was an ASIC - so they had the first ASIC 
back - and it basically worked first time! I take my hat off to people 
that can code chips like that.

Here we write FPGAs many times before we get it to work ...
>
> P.S. in theory our bootloader has a GDB stub talking the gdbserver
> protocol. I've never bothered to use it, but there are multiple
> solutions to a bunch of these problems, which we've used at various
> points as we needed them. There's been basically working jtag support in
> our SOC since late 2014, but it didn't turn out to be something we
> really needed. (I vaguely recall when we left off there was a single
> stepping bug, which didn't always stop at the right place. I don't
> remember if we've bothered to fix that yet because nobody's needed to
> use it.)
>
>>> I think the main problem is everybody tries to do DRAM init
>>> _through_the_jtag_, which is black magic at a level you really don't
>>> want to get on you if you can avoid it. Boards with buckets of SRAM are
>>> easier to bring up, but then you need dedicated multi-stage bootloaders
>>> with relocation code and you've basically added a second layer of mess...
>> DRAM init I agree isn't fun! Its why computers have BIOS, so the BIOS
>> does the hard stuff like DRAM init.
>>
>> At work, we had one board we couldn't get working with dram, so it had a
>> *whole* board of SRAM - just so we could get it to come up ...
> That ice40 stuff we were trying to get nvc to target to run j1 on is all
> sram. (When we say 'arduino country' that implies no dram controller.)
> And a lot of chips have a bunch of onboard sram even when they do have
> dram, so you can do your dram init from software. (Or NOR flash if you
> don't care about speed.) Heck, for years the reason you couldn't run
> u-boot under qemu is they refused to let you configure it NOT to do DRAM
> init, and although people eventually hacked their way around Wolfgang's
> persistent refusal/incomprehension:
>
> https://balau82.wordpress.com/2010/04/12/booting-linux-with-u-boot-on-qemu-arm/
>
> They still updated the u-boot FAQ entry saying "no, clearly not doing
> this is impossible because reasons, what's a QEMU?"
>
> https://www.denx.de/wiki/DULG/CanUBootBeConfiguredSuchThatItCanBeStartedInRAM
>
> As for j-core, we got dram init working long ago (there was an on-chip
> xilinx dram controller, then we wrote our own VHDL dram controller, then
> we wrote a _better_ one to replace the firs tone). So sure our designs
> have dram. Of course using it requires instantiating the j-core SOC on
> the board and running the boot ROM, which once again means your jtag is
> only useful in certain contexts unless you want to do a whole lot more
> work making it useful via duplicate infrastructure...
Doing your own dram controller was a good choice. It makes the route to 
ASIC easier - where you don't get a DRAM controller for free ...
> Once again: jtag's totally a side issue here.
>
>> Anyway JTAG is a bit like using a puppet, and you try kniting using a
>> puppet!
> I'm aware of the difficulty, yes. :)
>
> Rob

Regards,

David

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.j-core.org/pipermail/j-core/attachments/20170709/14b07884/attachment-0001.html>