[J-core] How the IRQ/timer stuff works.
Rob Landley
rob at landley.net
Mon Oct 28 22:48:12 UTC 2019
Here's the first message I sent back on August 8, and then a follow-up message below.
----------
Alright, looking at:
https://github.com/torvalds/linux/blob/master/drivers/irqchip/irq-jcore-aic.c
It looks like the initialization is:
#define JCORE_AIC_MAX_HWIRQ 127
#define JCORE_AIC1_MIN_HWIRQ 16
#define JCORE_AIC2_MIN_HWIRQ 64
#define JCORE_AIC1_INTPRI_REG 8
__raw_writel(0xffffffff, base + JCORE_AIC1_INTPRI_REG);
(The write is on line 85 of irq-jcore-aic.c.)
And the relevant bit of the device tree from
https://github.com/torvalds/linux/blob/master/arch/sh/boot/dts/j2_mimas_v2.dts
soc at abcd0000 {
compatible = "simple-bus";
ranges = <0 0xabcd0000 0x100000>;
#address-cells = <1>;
#size-cells = <1>;
aic: interrupt-controller at 200 {
compatible = "jcore,aic1";
reg = <0x200 0x10>;
interrupt-controller;
#interrupt-cells = <1>;
};
Which says base address 0xabcd0000 and then the interruput controller is at
offset 0x200 within that, length 0x10.
So I _think_ what it's doing to enable interrupts is just:
(unsigned *)(0xabcd0208) = 0xffffffff;
(I _think_ the interrupt-controller at 200 and reg = <0x200 are saying the same
thing and not adding to each other? If they're adding it would be 0xabcd0408 but
since _all_ the device tree segments are doing that I think it's just the same
thing in 2 places and not added. cache-controller at c0 and reg=<0xc0 adding would
be _silly_...)
Some relevant comments from the C file:
/*
* The J-Core AIC1 and AIC2 are cpu-local interrupt controllers and do
* not distinguish or use distinct irq number ranges for per-cpu event
* interrupts (timer, IPI). Since information to determine whether a
* particular irq number should be treated as per-cpu is not available
* at mapping time, we use a wrapper handler function which chooses
* the right handler at runtime based on whether IRQF_PERCPU was used
* when requesting the irq.
*/
* the only interrupt masking is at the cpu level and
* it affects all interrupts. We provide dummy mask/unmask. The hardware
* handles all interrupt control and clears pending status when the cpu
* accepts the interrupt.
*/
Now we need to enable the PIT so you _have_ an interrupt, and that's _this_ file:
https://github.com/torvalds/linux/blob/master/drivers/clocksource/jcore-pit.c
With this chunk of device tree just below the previous chunk in the dt file:
timer at 200 {
compatible = "jcore,pit";
reg = <0x200 0x30>;
interrupts = <0x48>;
};
So that says we're using IRQ number 0x48. The C file has a bunch more constants
at the top:
#define PIT_IRQ_SHIFT 12
#define PIT_PRIO_SHIFT 20
#define PIT_ENABLE_SHIFT 26
#define PIT_PRIO_MASK 0xf
#define REG_PITEN 0x00
#define REG_THROT 0x10
#define REG_COUNT 0x14
#define REG_BUSPD 0x18
#define REG_SECHI 0x20
#define REG_SECLO 0x24
#define REG_NSEC 0x28
And then jumping down to the init function, starting on line 158 of jcore-pit.c
we do:
err = clocksource_mmio_init(jcore_pit_base, "jcore_pit_cs",
NSEC_PER_SEC, 400, 32,
jcore_clocksource_read);
sched_clock_register(jcore_sched_clock_read, 32, NSEC_PER_SEC);
err = request_irq(pit_irq, jcore_timer_interrupt,
IRQF_TIMER | IRQF_PERCPU,
"jcore_pit", jcore_pit_percpu);
hwirq = irq_get_irq_data(pit_irq)->hwirq;
irqprio = (hwirq >> 2) & PIT_PRIO_MASK;
enable_val = (1U << PIT_ENABLE_SHIFT)
| (hwirq << PIT_IRQ_SHIFT)
| (irqprio << PIT_PRIO_SHIFT);
The guts of jcore_clocksource_read are:
static notrace u64 jcore_sched_clock_read(void)
{
u32 seclo, nsec, seclo0;
__iomem void *base = jcore_pit_base;
seclo = readl(base + REG_SECLO);
do {
seclo0 = seclo;
nsec = readl(base + REG_NSEC);
seclo = readl(base + REG_SECLO);
} while (seclo0 != seclo);
return seclo * NSEC_PER_SEC + nsec;
}
static u64 jcore_clocksource_read(struct clocksource *cs)
{
return jcore_sched_clock_read();
}
And then back in the init function there's per-cpu plumbing (do we have a PIT
per CPU?), which eventually calls:
cpuhp_setup_state(CPUHP_AP_JCORE_TIMER_STARTING,
"clockevents/jcore:starting",
jcore_pit_local_init, NULL);
And the guts of jcore_pit_local_init() are:
buspd = readl(pit->base + REG_BUSPD);
freq = DIV_ROUND_CLOSEST(NSEC_PER_SEC, buspd);
pit->periodic_delta = DIV_ROUND_CLOSEST(NSEC_PER_SEC, HZ * buspd);
clockevents_config_and_register(&pit->ced, freq, 1, ULONG_MAX);
And then here are the set/disable functions that actually seem to switch the PIT
on and off:
static int jcore_pit_disable(struct jcore_pit *pit)
{
writel(0, pit->base + REG_PITEN);
return 0;
}
static int jcore_pit_set(unsigned long delta, struct jcore_pit *pit)
{
jcore_pit_disable(pit);
writel(delta, pit->base + REG_THROT);
writel(pit->enable_val, pit->base + REG_PITEN);
return 0;
}
(Note: that enable_val is the one we worked out in the init function, it gets
marshalled into the structure in that percpu stuff I glossed over.)
And here's a BIG COMMENT in the file that's hopefully helpful:
/*
* The J-Core PIT is not hard-wired to a particular IRQ, but
* integrated with the interrupt controller such that the IRQ it
* generates is programmable, as follows:
*
* The bit layout of the PIT enable register is:
*
* .....e..ppppiiiiiiii............
*
* where the .'s indicate unrelated/unused bits, e is enable,
* p is priority, and i is hard irq number.
*
* For the PIT included in AIC1 (obsolete but still in use),
* any hard irq (trap number) can be programmed via the 8
* iiiiiiii bits, and a priority (0-15) is programmable
* separately in the pppp bits.
*
* For the PIT included in AIC2 (current), the programming
* interface is equivalent modulo interrupt mapping. This is
* why a different compatible tag was not used. However only
* traps 64-127 (the ones actually intended to be used for
* interrupts, rather than syscalls/exceptions/etc.) can be
* programmed (the high 2 bits of i are ignored) and the
* priority pppp is <<2'd and or'd onto the irq number. This
* choice seems to have been made on the hardware engineering
* side under an assumption that preserving old AIC1 priority
* mappings was important. Future models will likely ignore
* the pppp field.
*/
There's one important unanswered question, which is when an interrupt gets
called, where is the address of the function to jump to? There's generally an
array of pointers somewhere, but:
err = request_irq(pit_irq, jcore_timer_interrupt,
IRQF_TIMER | IRQF_PERCPU,
"jcore_pit", jcore_pit_percpu);
Kinda buries where that address gets written, I can try to dig it up if that's
helpful? I'm guessing it's in
https://www.renesas.com/us/en/doc/products/mpumcu/001/rej09b0171_superh.pdf
somewhere though?
Rob
-------- Forwarded Message --------
Subject: Re: It looks like my Mimas runs an AIC2 (the second version)???
Date: Thu, 8 Aug 2019 19:45:30 -0500
From: Rob Landley <rob at landley.net>
To: Joh-Tob Schäg <johtobsch at gmail.com>, D. Jeff Dionne <jeff at coresemi.io>
On 8/8/19 5:51 PM, Joh-Tob Schäg wrote:
> Rob in all his magic could dig out some old code for the AIC1.
I think it's for AIC2? I dug through the _current_ linux source (from torvalds'
github which is updated daily-ish), which I know runs on the bitstreams I sent
you. (Or did last month anyway.)
> Having access C-Code talking to both implementations now i think i can
> make that i run an AIC2.
> Please look over my arguments to see if i go wrong..
>
> Rob found the following information about the first (or so he thinks)
> AICs structure:
> #define REG_PITEN 0x00
> #define REG_THROT 0x10
> #define REG_COUNT 0x14
> #define REG_BUSPD 0x18
> #define REG_SECHI 0x20
> #define REG_SECLO 0x24
> #define REG_NSEC 0x28
>
> This leads me to the following "structure of an AIC1":
Note: AIC2 has some backwards compatibility stuff for AIC1. (And there was
something about sh2 vs sh3 interrupt vectors that Rich explained to me more than
once and I sadly don't remember. Kawasaki had strong opinions on it, but the sh3
version was used by sh4 so it's the direction we wanted to go in. Rich made both
work simultaneously with a Clever Hack...)
> struct iq_regs {
> uint32_t piten;
> uint32_t unknown1;
> uint16_t unknown2;
> uint32_t pit_throttle;
> uint32_t pit_counter;
> uint16_t bus_speed; // read-only
> uint32_t rtc_sec_hi;
> uint32_t rtc_sec_lo;
> uint32_t rtc_nsec;
> };
That seems unlikely: the alignment's off. Our hardware is going to put 32 bit
values at 4 byte addresses. If there are two uint16_t they'll be collated
together to take up a single 32 bit slot, so the others 32 bit values aren't
unaligned and straddling two aligned address slots.
Also, some access cares about word size: a 16 bit read does different things
than a 32 bit read. I don't _think_ any of ours is that crazy, but I've hit it
in a lot of I/O devices over the years. (Alignment, access granularity, access
order, and access _timing_ can all matter. Yes I've hit hardware where you had
to read registers in the right order and put delay loops between them for the
slow bus the hardware was on to clock the signal in to where it needed to be
before you could read it. The modern stuff should have a status bit you can spin
on, or will throw an iowait in there faking a pipeline stall or dram cache line
fetch or whatever mechanism it's got to stall the CPU until the data's ready...)
> I assume that that the base address of both AICs is 0xabcd0200. (this
> seems to be hinted at the device driver but is something that should
> be confirmed.)
Base address is 0xabcd0200 yes. Backwards compatibility with AIC1.
> This structure is clearly different from the one specified from the
> one in board.h (which i assume describes AIC2)
>
> struct aic_regs {
> uint32_t ctrl0;
> uint32_t brkadd;
> uint32_t ilevels;
> uint32_t ctrl1;
> uint32_t pit_throttle;
> uint32_t pit_counter;
> uint32_t clock_period; // read-only
> uint32_t ignore0;
> uint32_t rtc_sec_hi;
> uint32_t rtc_sec_lo;
> uint32_t rtc_nsec;
> };
See "alignment", yes. :)
Let's see, 11*4=44 bytes. The actual address range used by this is probably
going to be a power of 2, so there's 5 more 32 bit slots either unused or used
by something else (is that where the timers go?)
The earlier structure you had was 32 bytes, which makes more sense from a
hardware perspective (the VHDL is gonna have an "if high address bits =
0xabcd020x then giant block of stuff implementing the control lines") but your
alignment doesn't make sense because pit_throttle and pit_counter aren't aligned.
> When i run the following code:
> uint32_t i;
> for (i=0;i<1000000;i++){
> (DEVICE_GPIO)->value = (DEVICE_AIC0)->rtc_sec_lo;
> };
>
> (DEVICE_GPIO)->value = 15;
> wait_sec(2);
>
> for (i=0;i<1000000;i++){
> (DEVICE_GPIO)->value = (IQ)->rtc_sec_lo;
> };
Um... is this "struct iq_regs *IQ;" and 'struct aic_regs *DEVICE_AIC0;" ? You
didn't specify and it took me a while to figure that out...
> I get the behaviour of seeing the LEDs count up, then the 4 left most
> LEDs glowing for a moment, and then no LEDs glowing. (With a similar
> piece of code i could also confirm the the posistion of rtc_nsec
> favoring aic_regs over iq_res again.)
So you've confirmed rt_sec_lo lives at offset 44 in the structure, not offset 24
in the structure.
A fun macro, very useful:
http://man7.org/linux/man-pages/man3/offsetof.3.html
Shows you the offset of a structure member, in bytes.
struct potato {
char *blah, c, thing[2];
int walrus;
} source;
struct potato *p = &source;
int *s = (void *)(offsetof(struct potato, walrus)+(char *)p);
Now s points to walrus.
But it _also_ shows you whether the compiler inserted padding bytes and such,
which is very nice. Above, the int has to be 4 byte aligned, the char * is 4
bytes, c is 1 byte, thing[2] is 2 bytes, that's 7 total. It has to insert a byte
of padding to make walrus aligned...
Even if the compiler _doesn't_ insert padding (you can suppress it, for one
thing), the hardware may not accept unaligned addresses and will _silently_
fail. (Different ways on different hardware! Sometimes it rounds the address
down to the next aligned one. Sometimes the read is silently skipped. Sometimes
it triggers an exception interrupt. It all depends on what hardware you're
using. I think ours rounded down last I checked.)
Hardware that _does_ allow unaligned access tends to have FUN bugs, because
behind the scenes it's breaking the access into two accesses and bit shifting
them together in a hidden register, and if you think you're doing an atomic
access you're not, and if you're at the edge of a cache line and the first half
is in a page you can access and the second half is in a page you can't the
exception logic gets _insanely_ complicated and is often buggy... Some of
intel's spectre/meltdown nonsense involved that kind of thing over the years...
> This means a few things:
> - either my structure iq_regs is wrong
> - either my starting address is off
> - or i am running on the aic_regs
No, it means you've found where the seconds timer field is: (at 0xabcd022c) and
don't know where the others are yet. If the high seconds are right before it
they should be at 0xabcd028 and nanoseconds should be at 0xabcd030.
High seconds are basically always gonna be zero unless you set it, that takes
136 years to overflow; it's only there so we can set it to UTC and it tracks
from there. Remember, the y2038 problem is _signed_ 32 bit, unsigned has twice
as long to run, from jan 1 1970: 2038+68 = 2106.
But the nanosecond counter counts up once/second and if you send it to your LEDs
you should be able to watch the high bits toggle. The top bit should switch on
and then off again once per second (although it won't be quite even because
1000000000 isn't a power of 2. :)
> to my current understanding aic_regs is AIC2 but i am not sure about
> that, since the structure in "aic.vhd" described in the latest j-core
> release:
> -- Register layout for reads
> -- addr bits
> -- 5432 Contents
because 1 and 0, being the two "unaligned" bits, are ignored. I.E. these
addresses round down. :)
Tnis also means that if you treat aic as:
unsigned *aic = (unsigned *)0xabcd0200;
Then each aic[0] is pit_enable&friends, aic[1] is brkadd, aic[2] is ilevels...
> -----------------------------------
> -- 0000 - "00000" & pit_enable & count_enable & brk_enable &
> testvect & count
> -- 0001 - brkadd
> -- 0010 - ilevels
> -- 0011 - irq_i & q_irqs & pit_flag & db_ackcount & db_count
> -- 0100 - PIT throttle
> -- 0101 - PIT counter
> -- 0110 - Bus clock period in nanoseconds
> -- 0111 - zero
The first 8 entries in your unsigned integer array. Note: you can trust char to
be 1 byte, short to be 2 bytes, int to be 4 bytes, and long to be 32 bits on 32
bit systems and 64 bits on 64 bit systems, thanks to the LP64 standard which
Linux and BSD and MacOS all adhere to:
https://web.archive.org/web/20190406080023/http://www.unix.org/whitepapers/64bit.html
https://web.archive.org/web/20190406080016/http://www.unix.org/version2/whatsnew/lp64_wp.html
The only one that _doesn't_ do LP64 is Windows, and who cares about them
anymore. But here's why they chose to be uniquely broken.
https://web.archive.org/web/20080403224916/http://blogs.msdn.com/oldnewthing/archive/2005/01/31/363790.aspx
> -- 1x00 - RTC seconds upper 32 bits
> -- 1x01 - RTC seconds lower 32 bits
> -- 1x10 - RTC nanoseconds
> -- 1x11 - zero
The x here means "not actually reading that bit", so RTC seconds is there at
1001 and again at 1101. (So we have an unsigned aic[16] but the last 4 entries
are just copies of the previous 4 entries.) Translation: we have unsigned
aic[12] and know what each field does.
> If you guys can figure out which AIC iteration is described by
> aic_regs in board.h we know which version i am running.
The structure that had your timer at the right place sounds like the one your
board is using. And the longer one does seem to agree with what the VHDL says,
when interpreted properly.
And it might be easier to just treat this as an array of unsigned integers
rather than as a structure, because that way you don't have to worry about
alignment and structure packing and access granularity and such: the array is
_going_ to get it right. :) You can #define constants for the offsets
(RTC_EPOCH, RTC_SEC, RTC_NANO...) and thus:
printf("%u\n", clock[RTC_NANO]);
Here's the above VHDL table lightly cleaned up with decimal offsets:
> 0 0000 - "00000" & pit_enable & count_enable & brk_enable & testvect & count
> 1 0001 - brkadd
> 2 0010 - ilevels
> 3 0011 - irq_i & q_irqs & pit_flag & db_ackcount & db_count
> 4 0100 - PIT throttle
> 5 0101 - PIT counter
> 6 0110 - Bus clock period in nanoseconds
> 7 0111 - zero
> 8 1000 - RTC seconds upper 32 bits
> 9 1001 - RTC seconds lower 32 bits
> 10 1010 - RTC nanoseconds
> 11 1011 - zero
There's actually only 10 that matter because 2 are hardwired to 0, so:
#define AIC_CTRL1 0 // pit_enable, count_enable, brk_enable, testvect, count
#define AIC_BRKADD 1
#define AIC_ILEVELS 2
#define AIC_CTRL2 3 // irq_i, q_irqs, pit_flag, db_ackcount, db_count
#define AIC_PIT_THROT 4 // PIT throttle
#define AIC_PIT_CNT 5 // PIT counter
#define AIC_BUS_CLK 6 // bus clock period in nanoseconds
#define AIC_RTC_EPOCH 8 // RTC seconds upper 32 bits
#define AIC_RTC_SEC 9 // RTC seconds lower 32 bits
#define AIC_RTC_NANO 10 // RTC nanoseconds
And _really_, AIC is at 0xabcd0200 and RTC is at 0xabcd0220 so remove the last 3
from there and instead define a second pointer, and index it with:
#define RTC_EPOCH 0
#define RTC_SEC 1
#define RTC_NANO 2
> matches the structure described in aic_regs from board.h. What leaves
> my puzzled is that there is another file called "aic2.vhd" . Maybe
> "aic.vhd" describes the first version instead???
Could be there was a complete rewrite started and then we wound up just
modifying the original one instead of replacing it? Dunno, you'd have to ask
Niishi or Arakawa. (Or look in the repository history to see what the checkin
comments were when the file was created. There were some tight deadlines to get
demos out before tradeshows and such...)
Jeff said he had something like 22 hours of travel ahead of him and should next
be available sometime late saturday, but hopefully that gets you unblocked?
Now we have to work out how the AIC_CTRL1 and AIC_CTRL2 work because it looks
like lots of control bits packed together into the same field. (More VHDL
reading I expect...) Also, what are BRKADD, ILEVELS, and PIT_THROT? (I can see
PIT_CNT letting you know if you missed an interrupt?)
Rob
More information about the J-core
mailing list